2023-11-28T14:17:30.711 INFO:root:teuthology version: 0.0.1.dev209+g5ae09a5 2023-11-28T14:17:30.711 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi --owner scheduled_matan@teuthology --description crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} --archive /home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470178 -- /home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470178/orig.config.yaml 2023-11-28T14:17:30.755 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T14:17:30.840 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470178 branch: wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} email: null first_in_suite: false job_id: '7470178' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from sha1: 5537419ca37a48f0ea28d7872ae1f14ccf1a953b wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 5537419ca37a48f0ea28d7872ae1f14ccf1a953b thrashosds: min_in: 4 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: b9b6c42eed672399c3d01aa72f6cda1a665a3c35 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 9581 sha1: 5537419ca37a48f0ea28d7872ae1f14ccf1a953b sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: b9b6c42eed672399c3d01aa72f6cda1a665a3c35 targets: smithi055.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ4lVYdl6BZ/JhGhiPVEk5zjACkVuQ0SXB2blbo1W44KtcLD3nJ9YmK0QJZI9aMOQpZCZMGXhyZqff2YmxQGihM= smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMujrKC3zNd9bMVWC1Jc8fC/Rj6GOAlVisVeGkXaMfmsuFAl3j+cc/QV7CJ9l+/Jt1/vjKg46VKPe4FLB38t5m4= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_down: 0 chance_inject_pause_short: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - rados: balance_reads: true clients: - client.0 objects: 50 op_weights: copy_from: 0 delete: 50 read: 100 rollback: 0 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5ae09a5cc7064ca2bf994df19f3006b79794789a timestamp: 2023-11-28_14:04:23 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-28T14:17:30.840 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa; will attempt to use it 2023-11-28T14:17:30.841 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks 2023-11-28T14:17:30.841 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-28T14:17:30.844 INFO:teuthology.task.internal:Checking packages... 2023-11-28T14:17:30.868 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '5537419ca37a48f0ea28d7872ae1f14ccf1a953b' 2023-11-28T14:17:30.869 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-28T14:17:30.869 INFO:teuthology.packaging:ref: None 2023-11-28T14:17:30.869 INFO:teuthology.packaging:tag: None 2023-11-28T14:17:30.869 INFO:teuthology.packaging:branch: wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing 2023-11-28T14:17:30.869 INFO:teuthology.packaging:sha1: 5537419ca37a48f0ea28d7872ae1f14ccf1a953b 2023-11-28T14:17:30.869 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing 2023-11-28T14:17:31.011 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7503.g5537419c 2023-11-28T14:17:31.012 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-28T14:17:31.019 INFO:teuthology.task.internal:no buildpackages task found 2023-11-28T14:17:31.019 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-28T14:17:31.027 INFO:teuthology.task.internal:Saving configuration 2023-11-28T14:17:31.044 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-28T14:17:31.054 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T14:17:31.080 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470178', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 14:13:00.959084', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ4lVYdl6BZ/JhGhiPVEk5zjACkVuQ0SXB2blbo1W44KtcLD3nJ9YmK0QJZI9aMOQpZCZMGXhyZqff2YmxQGihM='} 2023-11-28T14:17:31.101 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470178', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 14:13:00.957845', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMujrKC3zNd9bMVWC1Jc8fC/Rj6GOAlVisVeGkXaMfmsuFAl3j+cc/QV7CJ9l+/Jt1/vjKg46VKPe4FLB38t5m4='} 2023-11-28T14:17:31.101 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-28T14:17:31.108 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2023-11-28T14:17:31.108 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-11-28T14:17:31.109 INFO:teuthology.run_tasks:Running task console_log... 2023-11-28T14:17:31.200 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa2153129d0>, signals=[15]) 2023-11-28T14:17:31.200 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-28T14:17:31.213 INFO:teuthology.task.internal:Opening connections... 2023-11-28T14:17:31.213 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2023-11-28T14:17:31.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:17:31.355 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2023-11-28T14:17:31.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:17:31.453 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-28T14:17:31.461 DEBUG:teuthology.orchestra.run.smithi055:> uname -m 2023-11-28T14:17:31.479 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2023-11-28T14:17:31.479 DEBUG:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2023-11-28T14:17:31.536 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="CentOS Stream" 2023-11-28T14:17:31.537 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="9" 2023-11-28T14:17:31.537 INFO:teuthology.orchestra.run.smithi055.stdout:ID="centos" 2023-11-28T14:17:31.537 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="rhel fedora" 2023-11-28T14:17:31.537 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="9" 2023-11-28T14:17:31.537 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el9" 2023-11-28T14:17:31.537 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-28T14:17:31.537 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2023-11-28T14:17:31.538 INFO:teuthology.orchestra.run.smithi055.stdout:LOGO="fedora-logo-icon" 2023-11-28T14:17:31.538 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-28T14:17:31.538 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://centos.org/" 2023-11-28T14:17:31.538 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-28T14:17:31.538 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-28T14:17:31.538 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-28T14:17:31.539 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2023-11-28T14:17:31.563 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2023-11-28T14:17:31.582 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2023-11-28T14:17:31.582 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2023-11-28T14:17:31.640 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Stream" 2023-11-28T14:17:31.640 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="9" 2023-11-28T14:17:31.640 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2023-11-28T14:17:31.640 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2023-11-28T14:17:31.640 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="9" 2023-11-28T14:17:31.640 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el9" 2023-11-28T14:17:31.641 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-28T14:17:31.641 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2023-11-28T14:17:31.641 INFO:teuthology.orchestra.run.smithi060.stdout:LOGO="fedora-logo-icon" 2023-11-28T14:17:31.641 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-28T14:17:31.641 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2023-11-28T14:17:31.641 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-28T14:17:31.641 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-28T14:17:31.641 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-28T14:17:31.641 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2023-11-28T14:17:31.680 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-28T14:17:31.695 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-28T14:17:31.707 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-28T14:17:31.708 DEBUG:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2023-11-28T14:17:31.711 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2023-11-28T14:17:31.728 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-28T14:17:31.740 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-28T14:17:31.741 DEBUG:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2023-11-28T14:17:31.769 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2023-11-28T14:17:31.797 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-28T14:17:31.861 INFO:teuthology.run_tasks:Running task kernel... 2023-11-28T14:17:31.883 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-28T14:17:31.883 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-28T14:17:31.884 DEBUG:teuthology.orchestra.run.smithi055:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T14:17:31.884 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T14:17:31.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T14:17:31.901 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2023-11-28T14:17:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T14:17:31.902 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2023-11-28T14:17:31.959 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-361.el9.x86_64 2023-11-28T14:17:31.960 INFO:teuthology.task.kernel:Running kernel on smithi055: 5.14.0-361.el9.x86_64 2023-11-28T14:17:31.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum install -y kernel 2023-11-28T14:17:31.961 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-361.el9.x86_64 2023-11-28T14:17:31.961 INFO:teuthology.task.kernel:Running kernel on smithi060: 5.14.0-361.el9.x86_64 2023-11-28T14:17:31.961 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum install -y kernel 2023-11-28T14:17:34.128 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2023-11-28T14:17:34.208 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2023-11-28T14:17:35.507 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2023-11-28T14:17:35.625 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2023-11-28T14:17:36.449 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 7.9 MB 00:00 2023-11-28T14:17:36.723 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 9.0 MB/s | 7.9 MB 00:00 2023-11-28T14:17:39.577 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 11 MB/s | 18 MB 00:01 2023-11-28T14:17:41.125 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 5.1 MB/s | 18 MB 00:03 2023-11-28T14:17:43.625 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.8 MB 00:00 2023-11-28T14:17:45.017 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 15 kB 00:00 2023-11-28T14:17:46.609 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 5.8 MB 00:01 2023-11-28T14:17:46.738 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2023-11-28T14:17:48.235 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2023-11-28T14:17:50.014 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2023-11-28T14:17:51.669 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2023-11-28T14:17:54.092 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-28T14:17:54.092 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-28T14:17:54.177 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:17:54.190 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:17:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2023-11-28T14:17:54.190 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:17:54.190 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:17:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: kernel x86_64 5.14.0-386.el9 baseos 5.5 M 2023-11-28T14:17:54.191 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2023-11-28T14:17:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core x86_64 5.14.0-386.el9 baseos 20 M 2023-11-28T14:17:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules x86_64 5.14.0-386.el9 baseos 38 M 2023-11-28T14:17:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core x86_64 5.14.0-386.el9 baseos 32 M 2023-11-28T14:17:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:17:54.192 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:17:54.192 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:17:54.192 INFO:teuthology.orchestra.run.smithi060.stdout:Install 4 Packages 2023-11-28T14:17:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:17:54.193 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 95 M 2023-11-28T14:17:54.193 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 121 M 2023-11-28T14:17:54.193 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:17:54.837 INFO:teuthology.orchestra.run.smithi060.stdout:(1/4): kernel-5.14.0-386.el9.x86_64.rpm 15 MB/s | 5.5 MB 00:00 2023-11-28T14:17:54.969 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-11-28T14:17:55.504 INFO:teuthology.orchestra.run.smithi060.stdout:(2/4): kernel-core-5.14.0-386.el9.x86_64.rpm 19 MB/s | 20 MB 00:01 2023-11-28T14:17:56.154 INFO:teuthology.orchestra.run.smithi060.stdout:(3/4): kernel-modules-5.14.0-386.el9.x86_64.rpm 22 MB/s | 38 MB 00:01 2023-11-28T14:17:56.812 INFO:teuthology.orchestra.run.smithi060.stdout:(4/4): kernel-modules-core-5.14.0-386.el9.x86_6 16 MB/s | 32 MB 00:01 2023-11-28T14:17:56.813 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:17:56.813 INFO:teuthology.orchestra.run.smithi060.stdout:Total 36 MB/s | 95 MB 00:02 2023-11-28T14:17:57.376 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-28T14:17:57.376 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-28T14:17:57.476 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:17:57.485 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:17:57.485 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-11-28T14:17:57.485 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:17:57.485 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:17:57.485 INFO:teuthology.orchestra.run.smithi055.stdout: kernel x86_64 5.14.0-386.el9 baseos 5.5 M 2023-11-28T14:17:57.486 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-28T14:17:57.486 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core x86_64 5.14.0-386.el9 baseos 20 M 2023-11-28T14:17:57.486 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules x86_64 5.14.0-386.el9 baseos 38 M 2023-11-28T14:17:57.486 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core x86_64 5.14.0-386.el9 baseos 32 M 2023-11-28T14:17:57.486 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:17:57.486 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:17:57.486 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:17:57.486 INFO:teuthology.orchestra.run.smithi055.stdout:Install 4 Packages 2023-11-28T14:17:57.486 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:17:57.487 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 95 M 2023-11-28T14:17:57.487 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 121 M 2023-11-28T14:17:57.487 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:17:57.698 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:17:58.132 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:17:58.132 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:17:59.454 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:17:59.455 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:18:02.231 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:18:02.919 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-core-5.14.0-386.el9.x86_64 1/4 2023-11-28T14:18:03.046 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-28T14:18:05.921 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-28T14:18:06.101 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-28T14:18:08.220 INFO:teuthology.orchestra.run.smithi055.stdout:(1/4): kernel-5.14.0-386.el9.x86_64.rpm 534 kB/s | 5.5 MB 00:10 2023-11-28T14:18:11.062 INFO:teuthology.orchestra.run.smithi055.stdout:(2/4): kernel-modules-core-5.14.0-386.el9.x86_6 11 MB/s | 32 MB 00:02 2023-11-28T14:18:11.519 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-28T14:18:11.519 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:18:16.232 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-core-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:18:18.486 INFO:teuthology.orchestra.run.smithi055.stdout:(3/4): kernel-core-5.14.0-386.el9.x86_64.rpm 996 kB/s | 20 MB 00:20 2023-11-28T14:18:22.020 INFO:teuthology.orchestra.run.smithi055.stdout:(4/4): kernel-modules-5.14.0-386.el9.x86_64.rpm 1.6 MB/s | 38 MB 00:24 2023-11-28T14:18:22.020 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:18:22.020 INFO:teuthology.orchestra.run.smithi055.stdout:Total 3.9 MB/s | 95 MB 00:24 2023-11-28T14:18:22.905 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:18:23.366 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:18:23.366 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:18:24.722 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:18:24.722 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:18:27.511 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:18:28.183 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-core-5.14.0-386.el9.x86_64 1/4 2023-11-28T14:18:28.312 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-28T14:18:31.203 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-28T14:18:31.391 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-28T14:18:36.348 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-28T14:18:36.466 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:18:41.453 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-core-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:18:46.899 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:18:46.944 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:18:47.361 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:18:47.361 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-5.14.0-386.el9.x86_64 1/4 2023-11-28T14:18:47.361 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-28T14:18:47.362 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-28T14:18:47.827 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-core-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:18:47.827 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:18:47.828 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:18:47.828 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-5.14.0-386.el9.x86_64 2023-11-28T14:18:47.828 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core-5.14.0-386.el9.x86_64 2023-11-28T14:18:47.828 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-5.14.0-386.el9.x86_64 2023-11-28T14:18:47.828 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core-5.14.0-386.el9.x86_64 2023-11-28T14:18:47.828 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:18:47.828 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:18:48.323 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2023-11-28T14:18:48.463 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-386.el9.x86_64 2023-11-28T14:18:48.464 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-386.el9.x86_64 2023-11-28T14:18:48.464 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-386.el9.x86_64 2023-11-28T14:18:48.464 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-28T14:18:48.465 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-28T14:18:48.465 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2023-11-28T14:18:48.465 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=None, version=distro) 2023-11-28T14:18:48.465 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2023-11-28T14:18:48.587 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-386.el9.x86_64 2023-11-28T14:18:48.588 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-386.el9.x86_64 2023-11-28T14:18:48.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo rpm -qi grub2-tools 2023-11-28T14:18:48.633 INFO:teuthology.orchestra.run.smithi060.stdout:Name : grub2-tools 2023-11-28T14:18:48.633 INFO:teuthology.orchestra.run.smithi060.stdout:Epoch : 1 2023-11-28T14:18:48.633 INFO:teuthology.orchestra.run.smithi060.stdout:Version : 2.06 2023-11-28T14:18:48.633 INFO:teuthology.orchestra.run.smithi060.stdout:Release : 68.el9 2023-11-28T14:18:48.633 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: x86_64 2023-11-28T14:18:48.634 INFO:teuthology.orchestra.run.smithi060.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-28T14:18:48.634 INFO:teuthology.orchestra.run.smithi060.stdout:Group : Unspecified 2023-11-28T14:18:48.634 INFO:teuthology.orchestra.run.smithi060.stdout:Size : 8264254 2023-11-28T14:18:48.634 INFO:teuthology.orchestra.run.smithi060.stdout:License : GPLv3+ 2023-11-28T14:18:48.634 INFO:teuthology.orchestra.run.smithi060.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-28T14:18:48.634 INFO:teuthology.orchestra.run.smithi060.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-28T14:18:48.634 INFO:teuthology.orchestra.run.smithi060.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-28T14:18:48.634 INFO:teuthology.orchestra.run.smithi060.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-28T14:18:48.634 INFO:teuthology.orchestra.run.smithi060.stdout:Packager : builder@centos.org 2023-11-28T14:18:48.634 INFO:teuthology.orchestra.run.smithi060.stdout:Vendor : CentOS 2023-11-28T14:18:48.634 INFO:teuthology.orchestra.run.smithi060.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-28T14:18:48.634 INFO:teuthology.orchestra.run.smithi060.stdout:Summary : Support tools for GRUB. 2023-11-28T14:18:48.635 INFO:teuthology.orchestra.run.smithi060.stdout:Description : 2023-11-28T14:18:48.635 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:18:48.635 INFO:teuthology.orchestra.run.smithi060.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-28T14:18:48.635 INFO:teuthology.orchestra.run.smithi060.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-28T14:18:48.635 INFO:teuthology.orchestra.run.smithi060.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-28T14:18:48.635 INFO:teuthology.orchestra.run.smithi060.stdout:hardware devices. 2023-11-28T14:18:48.635 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:18:48.635 INFO:teuthology.orchestra.run.smithi060.stdout:This subpackage provides tools for support of all platforms. 2023-11-28T14:18:48.636 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-28T14:18:48.636 INFO:teuthology.task.kernel:Updating grub on smithi060 to boot 5.14.0-386.el9.x86_64 2023-11-28T14:18:48.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-28T14:18:49.203 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2023-11-28T14:18:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-28T14:18:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:done 2023-11-28T14:18:50.773 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2023-11-28T14:18:50.839 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.6Ojr16If48 2023-11-28T14:18:50.839 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6Ojr16If48 2023-11-28T14:18:50.911 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.6Ojr16If48 2023-11-28T14:18:51.053 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.6Ojr16If48 is 6KB 2023-11-28T14:18:51.102 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.6Ojr16If48 2023-11-28T14:18:51.116 DEBUG:teuthology.orchestra.run.smithi060:> sudo /bin/ls /boot/loader/entries || true 2023-11-28T14:18:51.185 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-28T14:18:51.185 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-28T14:18:51.185 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-28T14:18:51.185 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-386.el9.x86_64.conf 2023-11-28T14:18:51.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-386.el9.x86_64 2023-11-28T14:18:51.326 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2023-11-28T14:19:12.288 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:19:12.335 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:19:12.771 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:19:12.772 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-5.14.0-386.el9.x86_64 1/4 2023-11-28T14:19:12.772 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-28T14:19:12.772 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-28T14:19:13.196 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-core-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:19:13.197 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:19:13.197 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:19:13.197 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-5.14.0-386.el9.x86_64 2023-11-28T14:19:13.197 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core-5.14.0-386.el9.x86_64 2023-11-28T14:19:13.197 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-5.14.0-386.el9.x86_64 2023-11-28T14:19:13.197 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core-5.14.0-386.el9.x86_64 2023-11-28T14:19:13.197 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:19:13.197 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:19:13.714 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2023-11-28T14:19:13.876 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-386.el9.x86_64 2023-11-28T14:19:13.876 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-386.el9.x86_64 2023-11-28T14:19:13.877 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-386.el9.x86_64 2023-11-28T14:19:13.877 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-28T14:19:13.877 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-28T14:19:13.877 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-28T14:19:13.877 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi055.front.sepia.ceph.com, path=None, version=distro) 2023-11-28T14:19:13.877 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2023-11-28T14:19:14.000 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-386.el9.x86_64 2023-11-28T14:19:14.001 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-386.el9.x86_64 2023-11-28T14:19:14.001 DEBUG:teuthology.orchestra.run.smithi055:> sudo rpm -qi grub2-tools 2023-11-28T14:19:14.039 INFO:teuthology.orchestra.run.smithi055.stdout:Name : grub2-tools 2023-11-28T14:19:14.039 INFO:teuthology.orchestra.run.smithi055.stdout:Epoch : 1 2023-11-28T14:19:14.039 INFO:teuthology.orchestra.run.smithi055.stdout:Version : 2.06 2023-11-28T14:19:14.040 INFO:teuthology.orchestra.run.smithi055.stdout:Release : 68.el9 2023-11-28T14:19:14.040 INFO:teuthology.orchestra.run.smithi055.stdout:Architecture: x86_64 2023-11-28T14:19:14.040 INFO:teuthology.orchestra.run.smithi055.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-28T14:19:14.040 INFO:teuthology.orchestra.run.smithi055.stdout:Group : Unspecified 2023-11-28T14:19:14.040 INFO:teuthology.orchestra.run.smithi055.stdout:Size : 8264254 2023-11-28T14:19:14.040 INFO:teuthology.orchestra.run.smithi055.stdout:License : GPLv3+ 2023-11-28T14:19:14.040 INFO:teuthology.orchestra.run.smithi055.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-28T14:19:14.040 INFO:teuthology.orchestra.run.smithi055.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-28T14:19:14.041 INFO:teuthology.orchestra.run.smithi055.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-28T14:19:14.041 INFO:teuthology.orchestra.run.smithi055.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-28T14:19:14.041 INFO:teuthology.orchestra.run.smithi055.stdout:Packager : builder@centos.org 2023-11-28T14:19:14.041 INFO:teuthology.orchestra.run.smithi055.stdout:Vendor : CentOS 2023-11-28T14:19:14.041 INFO:teuthology.orchestra.run.smithi055.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-28T14:19:14.041 INFO:teuthology.orchestra.run.smithi055.stdout:Summary : Support tools for GRUB. 2023-11-28T14:19:14.041 INFO:teuthology.orchestra.run.smithi055.stdout:Description : 2023-11-28T14:19:14.041 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:19:14.042 INFO:teuthology.orchestra.run.smithi055.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-28T14:19:14.042 INFO:teuthology.orchestra.run.smithi055.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-28T14:19:14.042 INFO:teuthology.orchestra.run.smithi055.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-28T14:19:14.042 INFO:teuthology.orchestra.run.smithi055.stdout:hardware devices. 2023-11-28T14:19:14.042 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:19:14.042 INFO:teuthology.orchestra.run.smithi055.stdout:This subpackage provides tools for support of all platforms. 2023-11-28T14:19:14.043 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-28T14:19:14.044 INFO:teuthology.task.kernel:Updating grub on smithi055 to boot 5.14.0-386.el9.x86_64 2023-11-28T14:19:14.044 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-28T14:19:14.692 INFO:teuthology.orchestra.run.smithi055.stderr:Generating grub configuration file ... 2023-11-28T14:19:16.100 INFO:teuthology.orchestra.run.smithi055.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-28T14:19:16.137 INFO:teuthology.orchestra.run.smithi055.stderr:done 2023-11-28T14:19:16.139 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2023-11-28T14:19:16.195 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.ATPP0iu5G6 2023-11-28T14:19:16.196 DEBUG:teuthology.orchestra.run.smithi055:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ATPP0iu5G6 2023-11-28T14:19:16.271 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.ATPP0iu5G6 2023-11-28T14:19:16.410 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.ATPP0iu5G6 is 6KB 2023-11-28T14:19:16.459 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.ATPP0iu5G6 2023-11-28T14:19:16.474 DEBUG:teuthology.orchestra.run.smithi055:> sudo /bin/ls /boot/loader/entries || true 2023-11-28T14:19:16.543 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-28T14:19:16.543 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-28T14:19:16.543 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-28T14:19:16.543 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-386.el9.x86_64.conf 2023-11-28T14:19:16.544 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-386.el9.x86_64 2023-11-28T14:19:16.698 DEBUG:teuthology.orchestra.run.smithi055:> sudo shutdown -r now 2023-11-28T14:19:21.333 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2023-11-28T14:19:21.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-11-28T14:19:21.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:19:39.746 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.60 2023-11-28T14:19:46.706 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-28T14:19:46.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-11-28T14:19:46.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:19:48.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-11-28T14:19:48.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:20:05.090 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.55 2023-11-28T14:20:14.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-11-28T14:20:14.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:20:46.004 DEBUG:teuthology.orchestra.run.smithi055:> true 2023-11-28T14:20:46.513 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-11-28T14:20:46.513 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-386.el9.x86_64"... 2023-11-28T14:20:46.513 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2023-11-28T14:20:46.528 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-386.el9.x86_64 2023-11-28T14:20:46.528 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-386.el9.x86_64 vs 5.14.0-386.el9.x86_64 2023-11-28T14:20:46.528 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-28T14:20:46.528 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-28T14:20:47.529 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-28T14:20:47.530 DEBUG:teuthology.orchestra.run.smithi055:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-28T14:20:47.633 INFO:teuthology.orchestra.run.smithi055.stdout:ttyS1 2023-11-28T14:20:47.673 DEBUG:teuthology.parallel:result is None 2023-11-28T14:20:48.753 DEBUG:teuthology.orchestra.remote:timed out 2023-11-28T14:21:00.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-11-28T14:21:00.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:21:01.118 DEBUG:teuthology.orchestra.run.smithi060:> true 2023-11-28T14:21:01.486 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-11-28T14:21:01.486 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-386.el9.x86_64"... 2023-11-28T14:21:01.486 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2023-11-28T14:21:01.502 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-386.el9.x86_64 2023-11-28T14:21:01.502 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-386.el9.x86_64 vs 5.14.0-386.el9.x86_64 2023-11-28T14:21:01.502 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-28T14:21:01.502 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-28T14:21:02.503 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2023-11-28T14:21:02.503 DEBUG:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-28T14:21:02.576 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2023-11-28T14:21:02.616 DEBUG:teuthology.parallel:result is None 2023-11-28T14:21:02.617 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-28T14:21:02.624 INFO:teuthology.task.internal:Creating test directory... 2023-11-28T14:21:02.624 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T14:21:02.627 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T14:21:02.646 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-28T14:21:02.675 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-28T14:21:02.685 INFO:teuthology.task.internal:Creating archive directory... 2023-11-28T14:21:02.685 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T14:21:02.689 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T14:21:02.728 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-28T14:21:02.735 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-28T14:21:02.735 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T14:21:02.766 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T14:21:02.801 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T14:21:02.807 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T14:21:02.820 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T14:21:02.825 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T14:21:02.827 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-28T14:21:02.834 INFO:teuthology.task.internal:Configuring sudo... 2023-11-28T14:21:02.834 DEBUG:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T14:21:02.864 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T14:21:02.901 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-28T14:21:02.911 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-28T14:21:02.911 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T14:21:02.939 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T14:21:02.958 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T14:21:03.040 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T14:21:03.094 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T14:21:03.180 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T14:21:03.227 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:21:03.227 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T14:21:03.300 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T14:21:03.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T14:21:03.405 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T14:21:03.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T14:21:03.499 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:21:03.499 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T14:21:03.572 DEBUG:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2023-11-28T14:21:03.575 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2023-11-28T14:21:03.628 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T14:21:03.661 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T14:21:04.030 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-28T14:21:04.039 INFO:teuthology.task.internal:Starting timer... 2023-11-28T14:21:04.039 INFO:teuthology.run_tasks:Running task pcp... 2023-11-28T14:21:04.049 INFO:teuthology.run_tasks:Running task selinux... 2023-11-28T14:21:04.057 DEBUG:teuthology.orchestra.run.smithi055:> sudo service auditd rotate 2023-11-28T14:21:04.138 INFO:teuthology.orchestra.run.smithi055.stdout:Rotating logs: 2023-11-28T14:21:04.140 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2023-11-28T14:21:04.188 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2023-11-28T14:21:04.190 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-28T14:21:04.190 DEBUG:teuthology.orchestra.run.smithi055:> /usr/sbin/getenforce 2023-11-28T14:21:04.232 INFO:teuthology.orchestra.run.smithi055.stdout:Permissive 2023-11-28T14:21:04.232 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2023-11-28T14:21:04.257 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2023-11-28T14:21:04.257 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi055.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi060.front.sepia.ceph.com': 'permissive'} 2023-11-28T14:21:04.258 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T14:21:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T14:21:04.307 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T14:21:04.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T14:21:04.339 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-28T14:21:04.340 DEBUG:teuthology.orchestra.run.smithi055:> sudo /usr/sbin/setenforce permissive 2023-11-28T14:21:04.384 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2023-11-28T14:21:04.426 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-28T14:21:04.551 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-28T14:21:04.551 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-28T14:21:04.568 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-28T14:21:04.569 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi055.front.sepia.ceph.com,smithi060.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-28T14:27:03.684 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi055.front.sepia.ceph.com'), Remote(name='ubuntu@smithi060.front.sepia.ceph.com')] 2023-11-28T14:27:03.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-11-28T14:27:03.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:27:03.786 DEBUG:teuthology.orchestra.run.smithi055:> true 2023-11-28T14:27:03.853 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-11-28T14:27:03.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-11-28T14:27:03.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:27:03.952 DEBUG:teuthology.orchestra.run.smithi060:> true 2023-11-28T14:27:04.021 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-11-28T14:27:04.021 INFO:teuthology.run_tasks:Running task clock... 2023-11-28T14:27:04.032 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-28T14:27:04.033 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T14:27:04.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T14:27:04.035 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T14:27:04.035 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T14:27:04.068 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-28T14:27:04.090 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-28T14:27:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-28T14:27:04.116 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-28T14:27:04.125 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: ntpd: command not found 2023-11-28T14:27:04.142 INFO:teuthology.orchestra.run.smithi055.stdout:506 Cannot talk to daemon 2023-11-28T14:27:04.152 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2023-11-28T14:27:04.162 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-28T14:27:04.169 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2023-11-28T14:27:04.182 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-28T14:27:04.188 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-28T14:27:04.208 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-28T14:27:04.229 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2023-11-28T14:27:04.232 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T14:27:04.233 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2023-11-28T14:27:04.233 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T14:27:04.233 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T14:27:04.233 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T14:27:04.233 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T14:27:04.255 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2023-11-28T14:27:04.259 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T14:27:04.259 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2023-11-28T14:27:04.259 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T14:27:04.259 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T14:27:04.259 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T14:27:04.259 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T14:27:04.259 INFO:teuthology.run_tasks:Running task install... 2023-11-28T14:27:04.268 DEBUG:teuthology.task.install:project ceph 2023-11-28T14:27:04.268 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b'}} 2023-11-28T14:27:04.268 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b'} 2023-11-28T14:27:04.268 INFO:teuthology.task.install:Using flavor: crimson 2023-11-28T14:27:04.274 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-28T14:27:04.274 INFO:teuthology.task.install:extra packages: [] 2023-11-28T14:27:04.274 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b', 'tag': None, 'wait_for_package': False} 2023-11-28T14:27:04.275 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=5537419ca37a48f0ea28d7872ae1f14ccf1a953b 2023-11-28T14:27:04.277 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b', 'tag': None, 'wait_for_package': False} 2023-11-28T14:27:04.277 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=5537419ca37a48f0ea28d7872ae1f14ccf1a953b 2023-11-28T14:27:04.425 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/centos/9/flavors/crimson/ 2023-11-28T14:27:04.425 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7503.g5537419c 2023-11-28T14:27:04.491 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/centos/9/flavors/crimson/ 2023-11-28T14:27:04.491 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7503.g5537419c 2023-11-28T14:27:04.542 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-28T14:27:04.543 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:27:04.543 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-28T14:27:04.575 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-28T14:27:04.575 DEBUG:teuthology.orchestra.run.smithi055:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-28T14:27:04.647 DEBUG:teuthology.orchestra.run.smithi055:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-28T14:27:04.658 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-28T14:27:04.659 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:27:04.659 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-28T14:27:04.692 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-28T14:27:04.692 DEBUG:teuthology.orchestra.run.smithi060:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-28T14:27:04.736 DEBUG:teuthology.orchestra.run.smithi055:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-28T14:27:04.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-28T14:27:04.765 INFO:teuthology.orchestra.run.smithi055.stdout:check_obsoletes = 1 2023-11-28T14:27:04.767 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum clean all 2023-11-28T14:27:04.852 DEBUG:teuthology.orchestra.run.smithi060:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-28T14:27:04.882 INFO:teuthology.orchestra.run.smithi060.stdout:check_obsoletes = 1 2023-11-28T14:27:04.883 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum clean all 2023-11-28T14:27:05.023 INFO:teuthology.orchestra.run.smithi055.stdout:73 files removed 2023-11-28T14:27:05.050 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-radosgw 2023-11-28T14:27:05.135 INFO:teuthology.orchestra.run.smithi060.stdout:73 files removed 2023-11-28T14:27:05.162 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw 2023-11-28T14:27:05.735 INFO:teuthology.orchestra.run.smithi055.stdout:ceph packages for x86_64 234 kB/s | 82 kB 00:00 2023-11-28T14:27:05.968 INFO:teuthology.orchestra.run.smithi060.stdout:ceph packages for x86_64 173 kB/s | 82 kB 00:00 2023-11-28T14:27:06.119 INFO:teuthology.orchestra.run.smithi055.stdout:ceph noarch packages 29 kB/s | 9.5 kB 00:00 2023-11-28T14:27:06.310 INFO:teuthology.orchestra.run.smithi060.stdout:ceph noarch packages 33 kB/s | 9.5 kB 00:00 2023-11-28T14:27:06.494 INFO:teuthology.orchestra.run.smithi055.stdout:ceph source packages 5.8 kB/s | 2.0 kB 00:00 2023-11-28T14:27:06.678 INFO:teuthology.orchestra.run.smithi060.stdout:ceph source packages 5.9 kB/s | 2.0 kB 00:00 2023-11-28T14:27:06.913 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2023-11-28T14:27:07.080 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 18 MB/s | 6.4 MB 00:00 2023-11-28T14:27:08.322 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2023-11-28T14:27:08.481 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2023-11-28T14:27:09.138 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 7.9 MB 00:00 2023-11-28T14:27:11.121 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 7.9 MB 00:02 2023-11-28T14:27:13.305 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 18 MB/s | 18 MB 00:01 2023-11-28T14:27:17.231 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 14 MB/s | 5.8 MB 00:00 2023-11-28T14:27:17.247 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 18 MB 00:06 2023-11-28T14:27:18.708 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 15 kB 00:00 2023-11-28T14:27:20.186 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2023-11-28T14:27:22.574 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 5.8 MB 00:01 2023-11-28T14:27:23.933 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 15 kB 00:00 2023-11-28T14:27:25.159 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2023-11-28T14:27:27.988 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:27:27.999 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:27:27.999 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-11-28T14:27:27.999 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:27:27.999 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:27:27.999 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw x86_64 2:18.0.0-7503.g5537419c.el9 ceph 8.4 M 2023-11-28T14:27:27.999 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2023-11-28T14:27:27.999 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 3.1 M 2023-11-28T14:27:27.999 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 3.0 M 2023-11-28T14:27:27.999 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2023-11-28T14:27:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options 2023-11-28T14:27:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-28T14:27:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base x86_64 2:18.0.0-7503.g5537419c.el9 ceph 6.6 M 2023-11-28T14:27:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common x86_64 2:18.0.0-7503.g5537419c.el9 ceph 17 M 2023-11-28T14:27:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux x86_64 2:18.0.0-7503.g5537419c.el9 ceph 24 k 2023-11-28T14:27:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-28T14:27:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-28T14:27:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-28T14:27:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 728 k 2023-11-28T14:27:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-28T14:27:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-28T14:27:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-28T14:27:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 470 k 2023-11-28T14:27:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-28T14:27:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 4.5 M 2023-11-28T14:27:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-28T14:27:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-28T14:27:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-28T14:27:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse 2023-11-28T14:27:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: x86_64 2:18.0.0-7503.g5537419c.el9 ceph 45 k 2023-11-28T14:27:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common x86_64 2:18.0.0-7503.g5537419c.el9 ceph 129 k 2023-11-28T14:27:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs x86_64 2:18.0.0-7503.g5537419c.el9 ceph 162 k 2023-11-28T14:27:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-28T14:27:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados x86_64 2:18.0.0-7503.g5537419c.el9 ceph 321 k 2023-11-28T14:27:28.002 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd x86_64 2:18.0.0-7503.g5537419c.el9 ceph 298 k 2023-11-28T14:27:28.002 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw x86_64 2:18.0.0-7503.g5537419c.el9 ceph 100 k 2023-11-28T14:27:28.002 INFO:teuthology.orchestra.run.smithi060.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-28T14:27:28.002 INFO:teuthology.orchestra.run.smithi060.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-28T14:27:28.002 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:27:28.002 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:27:28.002 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:27:28.002 INFO:teuthology.orchestra.run.smithi060.stdout:Install 27 Packages 2023-11-28T14:27:28.002 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 2 Packages 2023-11-28T14:27:28.002 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:27:28.003 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 54 M 2023-11-28T14:27:28.003 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:27:30.534 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 3.0 MB/s | 20 MB 00:06 2023-11-28T14:27:31.652 INFO:teuthology.orchestra.run.smithi060.stdout:(1/29): ceph-base-18.0.0-7503.g5537419c.el9.x86 1.8 MB/s | 6.6 MB 00:03 2023-11-28T14:27:31.746 INFO:teuthology.orchestra.run.smithi060.stdout:(2/29): ceph-selinux-18.0.0-7503.g5537419c.el9. 256 kB/s | 24 kB 00:00 2023-11-28T14:27:32.102 INFO:teuthology.orchestra.run.smithi060.stdout:(3/29): libcephfs2-18.0.0-7503.g5537419c.el9.x8 2.0 MB/s | 728 kB 00:00 2023-11-28T14:27:32.394 INFO:teuthology.orchestra.run.smithi060.stdout:(4/29): libradosstriper1-18.0.0-7503.g5537419c. 1.6 MB/s | 470 kB 00:00 2023-11-28T14:27:32.494 INFO:teuthology.orchestra.run.smithi060.stdout:(5/29): ceph-radosgw-18.0.0-7503.g5537419c.el9. 1.9 MB/s | 8.4 MB 00:04 2023-11-28T14:27:32.613 INFO:teuthology.orchestra.run.smithi060.stdout:(6/29): python3-ceph-argparse-18.0.0-7503.g5537 387 kB/s | 45 kB 00:00 2023-11-28T14:27:33.278 INFO:teuthology.orchestra.run.smithi060.stdout:(7/29): python3-ceph-common-18.0.0-7503.g553741 193 kB/s | 129 kB 00:00 2023-11-28T14:27:33.362 INFO:teuthology.orchestra.run.smithi060.stdout:(8/29): python3-cephfs-18.0.0-7503.g5537419c.el 1.9 MB/s | 162 kB 00:00 2023-11-28T14:27:33.545 INFO:teuthology.orchestra.run.smithi060.stdout:(9/29): python3-rados-18.0.0-7503.g5537419c.el9 1.7 MB/s | 321 kB 00:00 2023-11-28T14:27:33.721 INFO:teuthology.orchestra.run.smithi060.stdout:(10/29): python3-rbd-18.0.0-7503.g5537419c.el9. 1.7 MB/s | 298 kB 00:00 2023-11-28T14:27:33.821 INFO:teuthology.orchestra.run.smithi060.stdout:(11/29): python3-rgw-18.0.0-7503.g5537419c.el9. 999 kB/s | 100 kB 00:00 2023-11-28T14:27:33.921 INFO:teuthology.orchestra.run.smithi060.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2023-11-28T14:27:33.963 INFO:teuthology.orchestra.run.smithi060.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2023-11-28T14:27:33.997 INFO:teuthology.orchestra.run.smithi060.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2023-11-28T14:27:34.055 INFO:teuthology.orchestra.run.smithi060.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-11-28T14:27:34.098 INFO:teuthology.orchestra.run.smithi060.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2023-11-28T14:27:34.131 INFO:teuthology.orchestra.run.smithi060.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2023-11-28T14:27:34.306 INFO:teuthology.orchestra.run.smithi060.stdout:(18/29): librgw2-18.0.0-7503.g5537419c.el9.x86_ 2.3 MB/s | 4.5 MB 00:01 2023-11-28T14:27:34.413 INFO:teuthology.orchestra.run.smithi060.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2023-11-28T14:27:34.472 INFO:teuthology.orchestra.run.smithi060.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 417 kB/s | 24 kB 00:00 2023-11-28T14:27:34.530 INFO:teuthology.orchestra.run.smithi060.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 834 kB/s | 49 kB 00:00 2023-11-28T14:27:34.589 INFO:teuthology.orchestra.run.smithi060.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2023-11-28T14:27:34.715 INFO:teuthology.orchestra.run.smithi060.stdout:(23/29): libarrow-9.0.0-9.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2023-11-28T14:27:34.748 INFO:teuthology.orchestra.run.smithi060.stdout:(24/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 5.2 MB/s | 837 kB 00:00 2023-11-28T14:27:34.773 INFO:teuthology.orchestra.run.smithi060.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2023-11-28T14:27:34.891 INFO:teuthology.orchestra.run.smithi060.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2023-11-28T14:27:35.434 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-11-28T14:27:36.033 INFO:teuthology.orchestra.run.smithi060.stdout:(27/29): librados2-18.0.0-7503.g5537419c.el9.x8 2.5 MB/s | 3.1 MB 00:01 2023-11-28T14:27:36.241 INFO:teuthology.orchestra.run.smithi060.stdout:(28/29): librbd1-18.0.0-7503.g5537419c.el9.x86_ 2.2 MB/s | 3.0 MB 00:01 2023-11-28T14:27:38.262 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:27:38.278 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:27:38.278 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-28T14:27:38.279 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:27:38.279 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:27:38.279 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw x86_64 2:18.0.0-7503.g5537419c.el9 ceph 8.4 M 2023-11-28T14:27:38.279 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrading: 2023-11-28T14:27:38.279 INFO:teuthology.orchestra.run.smithi055.stdout: librados2 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 3.1 M 2023-11-28T14:27:38.279 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 3.0 M 2023-11-28T14:27:38.279 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-28T14:27:38.279 INFO:teuthology.orchestra.run.smithi055.stdout: boost-program-options 2023-11-28T14:27:38.279 INFO:teuthology.orchestra.run.smithi055.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-28T14:27:38.279 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base x86_64 2:18.0.0-7503.g5537419c.el9 ceph 6.6 M 2023-11-28T14:27:38.279 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common x86_64 2:18.0.0-7503.g5537419c.el9 ceph 17 M 2023-11-28T14:27:38.279 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux x86_64 2:18.0.0-7503.g5537419c.el9 ceph 24 k 2023-11-28T14:27:38.280 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-28T14:27:38.280 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-28T14:27:38.280 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-28T14:27:38.280 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 728 k 2023-11-28T14:27:38.280 INFO:teuthology.orchestra.run.smithi055.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-28T14:27:38.280 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-28T14:27:38.280 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-28T14:27:38.280 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 470 k 2023-11-28T14:27:38.280 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-28T14:27:38.280 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 4.5 M 2023-11-28T14:27:38.280 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-28T14:27:38.280 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-28T14:27:38.280 INFO:teuthology.orchestra.run.smithi055.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-28T14:27:38.281 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse 2023-11-28T14:27:38.281 INFO:teuthology.orchestra.run.smithi055.stdout: x86_64 2:18.0.0-7503.g5537419c.el9 ceph 45 k 2023-11-28T14:27:38.281 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common x86_64 2:18.0.0-7503.g5537419c.el9 ceph 129 k 2023-11-28T14:27:38.281 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs x86_64 2:18.0.0-7503.g5537419c.el9 ceph 162 k 2023-11-28T14:27:38.281 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-28T14:27:38.281 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados x86_64 2:18.0.0-7503.g5537419c.el9 ceph 321 k 2023-11-28T14:27:38.281 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd x86_64 2:18.0.0-7503.g5537419c.el9 ceph 298 k 2023-11-28T14:27:38.281 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw x86_64 2:18.0.0-7503.g5537419c.el9 ceph 100 k 2023-11-28T14:27:38.281 INFO:teuthology.orchestra.run.smithi055.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-28T14:27:38.281 INFO:teuthology.orchestra.run.smithi055.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-28T14:27:38.281 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:27:38.281 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:27:38.282 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:27:38.282 INFO:teuthology.orchestra.run.smithi055.stdout:Install 27 Packages 2023-11-28T14:27:38.282 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrade 2 Packages 2023-11-28T14:27:38.282 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:27:38.282 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 54 M 2023-11-28T14:27:38.282 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:27:41.519 INFO:teuthology.orchestra.run.smithi055.stdout:(1/29): ceph-radosgw-18.0.0-7503.g5537419c.el9. 2.9 MB/s | 8.4 MB 00:02 2023-11-28T14:27:41.610 INFO:teuthology.orchestra.run.smithi055.stdout:(2/29): ceph-selinux-18.0.0-7503.g5537419c.el9. 263 kB/s | 24 kB 00:00 2023-11-28T14:27:41.861 INFO:teuthology.orchestra.run.smithi055.stdout:(3/29): libcephfs2-18.0.0-7503.g5537419c.el9.x8 2.8 MB/s | 728 kB 00:00 2023-11-28T14:27:42.120 INFO:teuthology.orchestra.run.smithi055.stdout:(4/29): libradosstriper1-18.0.0-7503.g5537419c. 1.8 MB/s | 470 kB 00:00 2023-11-28T14:27:42.337 INFO:teuthology.orchestra.run.smithi055.stdout:(5/29): ceph-base-18.0.0-7503.g5537419c.el9.x86 1.8 MB/s | 6.6 MB 00:03 2023-11-28T14:27:42.349 INFO:teuthology.orchestra.run.smithi060.stdout:(29/29): ceph-common-18.0.0-7503.g5537419c.el9. 1.2 MB/s | 17 MB 00:14 2023-11-28T14:27:42.353 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:27:42.353 INFO:teuthology.orchestra.run.smithi060.stdout:Total 3.7 MB/s | 54 MB 00:14 2023-11-28T14:27:42.353 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:27:42.394 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:27:42.394 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:27:42.428 INFO:teuthology.orchestra.run.smithi055.stdout:(6/29): python3-ceph-argparse-18.0.0-7503.g5537 493 kB/s | 45 kB 00:00 2023-11-28T14:27:42.537 INFO:teuthology.orchestra.run.smithi055.stdout:(7/29): python3-ceph-common-18.0.0-7503.g553741 1.2 MB/s | 129 kB 00:00 2023-11-28T14:27:42.729 INFO:teuthology.orchestra.run.smithi055.stdout:(8/29): ceph-common-18.0.0-7503.g5537419c.el9.x 4.2 MB/s | 17 MB 00:04 2023-11-28T14:27:42.754 INFO:teuthology.orchestra.run.smithi055.stdout:(9/29): python3-cephfs-18.0.0-7503.g5537419c.el 744 kB/s | 162 kB 00:00 2023-11-28T14:27:42.805 INFO:teuthology.orchestra.run.smithi055.stdout:(10/29): python3-rados-18.0.0-7503.g5537419c.el 4.2 MB/s | 321 kB 00:00 2023-11-28T14:27:42.853 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:27:42.854 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:27:42.885 INFO:teuthology.orchestra.run.smithi055.stdout:(11/29): python3-rgw-18.0.0-7503.g5537419c.el9. 1.2 MB/s | 100 kB 00:00 2023-11-28T14:27:42.910 INFO:teuthology.orchestra.run.smithi055.stdout:(12/29): python3-rbd-18.0.0-7503.g5537419c.el9. 1.9 MB/s | 298 kB 00:00 2023-11-28T14:27:43.027 INFO:teuthology.orchestra.run.smithi055.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 745 kB/s | 106 kB 00:00 2023-11-28T14:27:43.061 INFO:teuthology.orchestra.run.smithi055.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2023-11-28T14:27:43.086 INFO:teuthology.orchestra.run.smithi055.stdout:(15/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2023-11-28T14:27:43.137 INFO:teuthology.orchestra.run.smithi055.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2023-11-28T14:27:43.162 INFO:teuthology.orchestra.run.smithi055.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-11-28T14:27:43.179 INFO:teuthology.orchestra.run.smithi055.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 990 kB/s | 41 kB 00:00 2023-11-28T14:27:43.388 INFO:teuthology.orchestra.run.smithi055.stdout:(19/29): librgw2-18.0.0-7503.g5537419c.el9.x86_ 3.5 MB/s | 4.5 MB 00:01 2023-11-28T14:27:43.746 INFO:teuthology.orchestra.run.smithi055.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 69 kB/s | 24 kB 00:00 2023-11-28T14:27:43.838 INFO:teuthology.orchestra.run.smithi055.stdout:(21/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 457 kB/s | 309 kB 00:00 2023-11-28T14:27:43.855 INFO:teuthology.orchestra.run.smithi055.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2023-11-28T14:27:43.947 INFO:teuthology.orchestra.run.smithi055.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2023-11-28T14:27:44.106 INFO:teuthology.orchestra.run.smithi055.stdout:(24/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.3 MB/s | 837 kB 00:00 2023-11-28T14:27:44.131 INFO:teuthology.orchestra.run.smithi055.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2023-11-28T14:27:44.198 INFO:teuthology.orchestra.run.smithi055.stdout:(26/29): libarrow-9.0.0-9.el9.x86_64.rpm 4.3 MB/s | 4.4 MB 00:01 2023-11-28T14:27:44.278 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:27:44.316 INFO:teuthology.orchestra.run.smithi055.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 7.7 MB/s | 1.6 MB 00:00 2023-11-28T14:27:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-11-28T14:27:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librados2-2:18.0.0-7503.g5537419c.el9.x86_64 2/31 2023-11-28T14:27:44.857 INFO:teuthology.orchestra.run.smithi055.stdout:(28/29): librbd1-18.0.0-7503.g5537419c.el9.x86_ 4.6 MB/s | 3.0 MB 00:00 2023-11-28T14:27:44.919 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:18.0.0-7503.g5537419c.el9.x86_64 2/31 2023-11-28T14:27:45.018 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-2:18.0.0-7503.g5537419c.el9.x86_64 3/31 2023-11-28T14:27:45.044 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 4/31 2023-11-28T14:27:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 4/31 2023-11-28T14:27:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2023-11-28T14:27:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2023-11-28T14:27:45.195 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2023-11-28T14:27:45.239 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-argparse-2:18.0.0-7503.g5537419c.el9. 8/31 2023-11-28T14:27:45.249 INFO:teuthology.orchestra.run.smithi055.stdout:(29/29): librados2-18.0.0-7503.g5537419c.el9.x8 2.8 MB/s | 3.1 MB 00:01 2023-11-28T14:27:45.255 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:27:45.255 INFO:teuthology.orchestra.run.smithi055.stdout:Total 7.7 MB/s | 54 MB 00:06 2023-11-28T14:27:45.256 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:27:45.296 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:27:45.297 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:27:45.308 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-2:18.0.0-7503.g5537419c.el9.x86_64 9/31 2023-11-28T14:27:45.331 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradosstriper1-2:18.0.0-7503.g5537419c.el9.x86_6 10/31 2023-11-28T14:27:45.403 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7503.g5537419c.el9.x86_6 10/31 2023-11-28T14:27:45.438 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2023-11-28T14:27:45.550 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-11-28T14:27:45.600 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-11-28T14:27:45.639 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-11-28T14:27:45.671 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-11-28T14:27:45.700 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-11-28T14:27:45.728 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-11-28T14:27:45.763 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:27:45.764 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:27:46.061 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-11-28T14:27:46.091 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 19/31 2023-11-28T14:27:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 19/31 2023-11-28T14:27:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-2:18.0.0-7503.g5537419c.el9.x86_64 20/31 2023-11-28T14:27:46.249 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-11-28T14:27:46.704 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-11-28T14:27:46.735 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librgw2-2:18.0.0-7503.g5537419c.el9.x86_64 23/31 2023-11-28T14:27:46.790 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librgw2-2:18.0.0-7503.g5537419c.el9.x86_64 23/31 2023-11-28T14:27:46.868 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-2:18.0.0-7503.g5537419c.el9.x86_64 24/31 2023-11-28T14:27:46.904 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-common-2:18.0.0-7503.g5537419c.el9.x8 25/31 2023-11-28T14:27:47.157 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:27:47.538 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-11-28T14:27:47.602 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librados2-2:18.0.0-7503.g5537419c.el9.x86_64 2/31 2023-11-28T14:27:47.677 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:18.0.0-7503.g5537419c.el9.x86_64 2/31 2023-11-28T14:27:47.769 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rados-2:18.0.0-7503.g5537419c.el9.x86_64 3/31 2023-11-28T14:27:47.803 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 4/31 2023-11-28T14:27:47.887 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 4/31 2023-11-28T14:27:47.915 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2023-11-28T14:27:47.941 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2023-11-28T14:27:47.971 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2023-11-28T14:27:48.015 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-argparse-2:18.0.0-7503.g5537419c.el9. 8/31 2023-11-28T14:27:48.084 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cephfs-2:18.0.0-7503.g5537419c.el9.x86_64 9/31 2023-11-28T14:27:48.107 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libradosstriper1-2:18.0.0-7503.g5537419c.el9.x86_6 10/31 2023-11-28T14:27:48.196 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7503.g5537419c.el9.x86_6 10/31 2023-11-28T14:27:48.231 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2023-11-28T14:27:48.342 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-11-28T14:27:48.375 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-11-28T14:27:48.414 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-11-28T14:27:48.453 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-11-28T14:27:48.482 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-11-28T14:27:48.515 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-11-28T14:27:48.850 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-11-28T14:27:48.878 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 19/31 2023-11-28T14:27:48.968 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 19/31 2023-11-28T14:27:48.999 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rbd-2:18.0.0-7503.g5537419c.el9.x86_64 20/31 2023-11-28T14:27:49.036 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-11-28T14:27:49.057 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 26/31 2023-11-28T14:27:49.101 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 26/31 2023-11-28T14:27:49.490 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-11-28T14:27:49.514 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librgw2-2:18.0.0-7503.g5537419c.el9.x86_64 23/31 2023-11-28T14:27:49.585 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librgw2-2:18.0.0-7503.g5537419c.el9.x86_64 23/31 2023-11-28T14:27:49.664 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rgw-2:18.0.0-7503.g5537419c.el9.x86_64 24/31 2023-11-28T14:27:49.708 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-common-2:18.0.0-7503.g5537419c.el9.x8 25/31 2023-11-28T14:27:49.893 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 26/31 2023-11-28T14:27:49.938 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 27/31 2023-11-28T14:27:50.010 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 27/31 2023-11-28T14:27:50.010 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-28T14:27:50.010 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-28T14:27:50.010 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:27:50.048 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-selinux-2:18.0.0-7503.g5537419c.el9.x86_64 28/31 2023-11-28T14:27:51.791 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 26/31 2023-11-28T14:27:51.821 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 26/31 2023-11-28T14:27:52.611 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 26/31 2023-11-28T14:27:52.666 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 27/31 2023-11-28T14:27:52.753 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 27/31 2023-11-28T14:27:52.753 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-28T14:27:52.753 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-28T14:27:52.753 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:27:52.800 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-selinux-2:18.0.0-7503.g5537419c.el9.x86_64 28/31 2023-11-28T14:28:00.131 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7503.g5537419c.el9.x86_64 28/31 2023-11-28T14:28:00.131 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /sys 2023-11-28T14:28:00.131 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /proc 2023-11-28T14:28:00.131 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /mnt 2023-11-28T14:28:00.132 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /var/tmp 2023-11-28T14:28:00.132 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /home 2023-11-28T14:28:00.132 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /root 2023-11-28T14:28:00.132 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /tmp 2023-11-28T14:28:00.132 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:28:01.006 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-2:18.0.0-7503.g5537419c.el9.x86_64 29/31 2023-11-28T14:28:01.032 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7503.g5537419c.el9.x86_64 29/31 2023-11-28T14:28:01.032 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:28:01.032 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-28T14:28:01.032 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T14:28:01.033 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T14:28:01.033 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:28:01.034 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-28T14:28:01.108 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-28T14:28:01.108 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-28T14:28:02.340 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-28T14:28:02.340 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 1/31 2023-11-28T14:28:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 2/31 2023-11-28T14:28:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-2:18.0.0-7503.g5537419c.el9.x86_64 3/31 2023-11-28T14:28:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-selinux-2:18.0.0-7503.g5537419c.el9.x86_64 4/31 2023-11-28T14:28:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 5/31 2023-11-28T14:28:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradosstriper1-2:18.0.0-7503.g5537419c.el9.x86_6 6/31 2023-11-28T14:28:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librgw2-2:18.0.0-7503.g5537419c.el9.x86_64 7/31 2023-11-28T14:28:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7503.g5537419c.el9. 8/31 2023-11-28T14:28:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-common-2:18.0.0-7503.g5537419c.el9.x8 9/31 2023-11-28T14:28:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-2:18.0.0-7503.g5537419c.el9.x86_64 10/31 2023-11-28T14:28:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-2:18.0.0-7503.g5537419c.el9.x86_64 11/31 2023-11-28T14:28:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-2:18.0.0-7503.g5537419c.el9.x86_64 12/31 2023-11-28T14:28:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-2:18.0.0-7503.g5537419c.el9.x86_64 13/31 2023-11-28T14:28:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-11-28T14:28:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-11-28T14:28:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-11-28T14:28:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-11-28T14:28:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-11-28T14:28:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-11-28T14:28:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-11-28T14:28:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-11-28T14:28:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-11-28T14:28:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-11-28T14:28:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-11-28T14:28:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-11-28T14:28:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-11-28T14:28:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-11-28T14:28:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:18.0.0-7503.g5537419c.el9.x86_64 28/31 2023-11-28T14:28:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-11-28T14:28:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 30/31 2023-11-28T14:28:02.998 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7503.g5537419c.el9.x86_64 28/31 2023-11-28T14:28:02.998 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /sys 2023-11-28T14:28:02.998 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /proc 2023-11-28T14:28:02.998 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /mnt 2023-11-28T14:28:02.998 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /var/tmp 2023-11-28T14:28:02.998 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /home 2023-11-28T14:28:02.998 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /root 2023-11-28T14:28:02.998 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /tmp 2023-11-28T14:28:02.999 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:28:03.279 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-11-28T14:28:03.279 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:28:03.279 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2023-11-28T14:28:03.279 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.279 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.279 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:28:03.279 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-28T14:28:03.279 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.279 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.279 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.280 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.280 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-28T14:28:03.280 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-28T14:28:03.280 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-28T14:28:03.280 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.280 INFO:teuthology.orchestra.run.smithi060.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-28T14:28:03.280 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-28T14:28:03.280 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-28T14:28:03.280 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.280 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-28T14:28:03.280 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.280 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-28T14:28:03.281 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-28T14:28:03.281 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-28T14:28:03.281 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.281 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.281 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.281 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-28T14:28:03.281 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.281 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.281 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:03.281 INFO:teuthology.orchestra.run.smithi060.stdout: re2-1:20211101-3.el9.x86_64 2023-11-28T14:28:03.281 INFO:teuthology.orchestra.run.smithi060.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-28T14:28:03.282 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:28:03.282 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:28:03.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-test 2023-11-28T14:28:03.874 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-radosgw-2:18.0.0-7503.g5537419c.el9.x86_64 29/31 2023-11-28T14:28:03.900 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7503.g5537419c.el9.x86_64 29/31 2023-11-28T14:28:03.900 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:28:03.900 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-28T14:28:03.900 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T14:28:03.900 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T14:28:03.900 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:28:03.902 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-28T14:28:04.110 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-28T14:28:04.110 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-28T14:28:04.237 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:39 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:28:04.389 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:28:04.390 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:28:04.390 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-11-28T14:28:04.391 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:28:04.391 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:28:04.391 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test x86_64 2:18.0.0-7503.g5537419c.el9 ceph 117 M 2023-11-28T14:28:04.391 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2023-11-28T14:28:04.391 INFO:teuthology.orchestra.run.smithi060.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-11-28T14:28:04.391 INFO:teuthology.orchestra.run.smithi060.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-28T14:28:04.391 INFO:teuthology.orchestra.run.smithi060.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-11-28T14:28:04.391 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite x86_64 2:18.0.0-7503.g5537419c.el9 ceph 158 k 2023-11-28T14:28:04.391 INFO:teuthology.orchestra.run.smithi060.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-11-28T14:28:04.391 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-28T14:28:04.391 INFO:teuthology.orchestra.run.smithi060.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-28T14:28:04.391 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-28T14:28:04.392 INFO:teuthology.orchestra.run.smithi060.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-11-28T14:28:04.392 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:28:04.392 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:28:04.392 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:28:04.392 INFO:teuthology.orchestra.run.smithi060.stdout:Install 10 Packages 2023-11-28T14:28:04.392 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:28:04.392 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 120 M 2023-11-28T14:28:04.393 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 1.1 G 2023-11-28T14:28:04.393 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:28:04.929 INFO:teuthology.orchestra.run.smithi060.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 186 kB 00:00 2023-11-28T14:28:05.095 INFO:teuthology.orchestra.run.smithi060.stdout:(2/10): libasan-11.4.1-2.3.el9.x86_64.rpm 2.5 MB/s | 418 kB 00:00 2023-11-28T14:28:05.137 INFO:teuthology.orchestra.run.smithi060.stdout:(3/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2023-11-28T14:28:05.179 INFO:teuthology.orchestra.run.smithi060.stdout:(4/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-11-28T14:28:05.229 INFO:teuthology.orchestra.run.smithi060.stdout:(5/10): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2023-11-28T14:28:05.263 INFO:teuthology.orchestra.run.smithi060.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-11-28T14:28:05.300 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-28T14:28:05.300 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 1/31 2023-11-28T14:28:05.300 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 2/31 2023-11-28T14:28:05.300 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-radosgw-2:18.0.0-7503.g5537419c.el9.x86_64 3/31 2023-11-28T14:28:05.300 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-selinux-2:18.0.0-7503.g5537419c.el9.x86_64 4/31 2023-11-28T14:28:05.300 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 5/31 2023-11-28T14:28:05.301 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libradosstriper1-2:18.0.0-7503.g5537419c.el9.x86_6 6/31 2023-11-28T14:28:05.301 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librgw2-2:18.0.0-7503.g5537419c.el9.x86_64 7/31 2023-11-28T14:28:05.301 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7503.g5537419c.el9. 8/31 2023-11-28T14:28:05.301 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-common-2:18.0.0-7503.g5537419c.el9.x8 9/31 2023-11-28T14:28:05.301 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cephfs-2:18.0.0-7503.g5537419c.el9.x86_64 10/31 2023-11-28T14:28:05.301 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rados-2:18.0.0-7503.g5537419c.el9.x86_64 11/31 2023-11-28T14:28:05.301 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rbd-2:18.0.0-7503.g5537419c.el9.x86_64 12/31 2023-11-28T14:28:05.301 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rgw-2:18.0.0-7503.g5537419c.el9.x86_64 13/31 2023-11-28T14:28:05.301 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-11-28T14:28:05.301 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-11-28T14:28:05.301 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-11-28T14:28:05.301 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-11-28T14:28:05.302 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-11-28T14:28:05.302 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-11-28T14:28:05.302 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-11-28T14:28:05.302 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-11-28T14:28:05.302 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-11-28T14:28:05.302 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-11-28T14:28:05.302 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-11-28T14:28:05.302 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-11-28T14:28:05.302 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-11-28T14:28:05.302 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-11-28T14:28:05.302 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:18.0.0-7503.g5537419c.el9.x86_64 28/31 2023-11-28T14:28:05.303 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-11-28T14:28:05.303 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 30/31 2023-11-28T14:28:05.404 INFO:teuthology.orchestra.run.smithi060.stdout:(7/10): libcephsqlite-18.0.0-7503.g5537419c.el9 242 kB/s | 158 kB 00:00 2023-11-28T14:28:05.580 INFO:teuthology.orchestra.run.smithi060.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 695 kB/s | 122 kB 00:00 2023-11-28T14:28:05.738 INFO:teuthology.orchestra.run.smithi060.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 2.6 MB/s | 1.3 MB 00:00 2023-11-28T14:28:06.359 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-11-28T14:28:06.359 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:28:06.359 INFO:teuthology.orchestra.run.smithi055.stdout:Upgraded: 2023-11-28T14:28:06.359 INFO:teuthology.orchestra.run.smithi055.stdout: librados2-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.359 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.359 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:28:06.360 INFO:teuthology.orchestra.run.smithi055.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-28T14:28:06.360 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.360 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.360 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.360 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.360 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-28T14:28:06.360 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-28T14:28:06.360 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-28T14:28:06.360 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.360 INFO:teuthology.orchestra.run.smithi055.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-28T14:28:06.360 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-28T14:28:06.361 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-28T14:28:06.361 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.361 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-28T14:28:06.361 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.361 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-28T14:28:06.361 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-28T14:28:06.361 INFO:teuthology.orchestra.run.smithi055.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-28T14:28:06.361 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.361 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.361 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.361 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-28T14:28:06.361 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.362 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.362 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:06.362 INFO:teuthology.orchestra.run.smithi055.stdout: re2-1:20211101-3.el9.x86_64 2023-11-28T14:28:06.362 INFO:teuthology.orchestra.run.smithi055.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-28T14:28:06.362 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:28:06.362 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:28:06.740 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-test 2023-11-28T14:28:07.338 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:32 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:28:07.494 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:28:07.495 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:28:07.495 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-28T14:28:07.496 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:28:07.496 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:28:07.496 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test x86_64 2:18.0.0-7503.g5537419c.el9 ceph 117 M 2023-11-28T14:28:07.496 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-28T14:28:07.496 INFO:teuthology.orchestra.run.smithi055.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-11-28T14:28:07.496 INFO:teuthology.orchestra.run.smithi055.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-28T14:28:07.496 INFO:teuthology.orchestra.run.smithi055.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-11-28T14:28:07.496 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite x86_64 2:18.0.0-7503.g5537419c.el9 ceph 158 k 2023-11-28T14:28:07.496 INFO:teuthology.orchestra.run.smithi055.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-11-28T14:28:07.496 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-28T14:28:07.496 INFO:teuthology.orchestra.run.smithi055.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-28T14:28:07.496 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-28T14:28:07.496 INFO:teuthology.orchestra.run.smithi055.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-11-28T14:28:07.497 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:28:07.497 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:28:07.497 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:28:07.497 INFO:teuthology.orchestra.run.smithi055.stdout:Install 10 Packages 2023-11-28T14:28:07.497 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:28:07.497 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 120 M 2023-11-28T14:28:07.498 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 1.1 G 2023-11-28T14:28:07.498 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:28:08.148 INFO:teuthology.orchestra.run.smithi055.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 655 kB/s | 186 kB 00:00 2023-11-28T14:28:08.207 INFO:teuthology.orchestra.run.smithi055.stdout:(2/10): libasan-11.4.1-2.3.el9.x86_64.rpm 7.1 MB/s | 418 kB 00:00 2023-11-28T14:28:08.274 INFO:teuthology.orchestra.run.smithi055.stdout:(3/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 2.7 MB/s | 186 kB 00:00 2023-11-28T14:28:08.316 INFO:teuthology.orchestra.run.smithi055.stdout:(4/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-11-28T14:28:08.358 INFO:teuthology.orchestra.run.smithi055.stdout:(5/10): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2023-11-28T14:28:08.392 INFO:teuthology.orchestra.run.smithi055.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-11-28T14:28:08.442 INFO:teuthology.orchestra.run.smithi055.stdout:(7/10): libcephsqlite-18.0.0-7503.g5537419c.el9 274 kB/s | 158 kB 00:00 2023-11-28T14:28:08.859 INFO:teuthology.orchestra.run.smithi055.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 292 kB/s | 122 kB 00:00 2023-11-28T14:28:09.059 INFO:teuthology.orchestra.run.smithi055.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 1.9 MB/s | 1.3 MB 00:00 2023-11-28T14:28:18.983 INFO:teuthology.orchestra.run.smithi055.stdout:(10/10): ceph-test-18.0.0-7503.g5537419c.el9.x8 11 MB/s | 117 MB 00:11 2023-11-28T14:28:18.986 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:28:18.986 INFO:teuthology.orchestra.run.smithi055.stdout:Total 10 MB/s | 120 MB 00:11 2023-11-28T14:28:18.987 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:28:19.011 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:28:19.011 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:28:19.863 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:28:19.863 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:28:21.015 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:28:21.115 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-11-28T14:28:21.150 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-11-28T14:28:21.195 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-11-28T14:28:21.234 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-11-28T14:28:21.267 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-11-28T14:28:21.299 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : jq-1.6-16.el9.x86_64 6/10 2023-11-28T14:28:21.337 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-11-28T14:28:21.377 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-11-28T14:28:21.413 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephsqlite-2:18.0.0-7503.g5537419c.el9.x86_64 9/10 2023-11-28T14:28:37.044 INFO:teuthology.orchestra.run.smithi060.stdout:(10/10): ceph-test-18.0.0-7503.g5537419c.el9.x8 3.6 MB/s | 117 MB 00:32 2023-11-28T14:28:37.049 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:28:37.049 INFO:teuthology.orchestra.run.smithi060.stdout:Total 3.7 MB/s | 120 MB 00:32 2023-11-28T14:28:37.049 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:28:37.073 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:28:37.073 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:28:37.927 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:28:37.927 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:28:38.637 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7503.g5537419c.el9.x86_64 9/10 2023-11-28T14:28:38.714 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-test-2:18.0.0-7503.g5537419c.el9.x86_64 10/10 2023-11-28T14:28:39.219 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:28:39.311 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-11-28T14:28:39.343 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-11-28T14:28:39.407 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-11-28T14:28:39.450 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-11-28T14:28:39.486 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-11-28T14:28:39.517 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : jq-1.6-16.el9.x86_64 6/10 2023-11-28T14:28:39.558 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-11-28T14:28:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-11-28T14:28:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephsqlite-2:18.0.0-7503.g5537419c.el9.x86_64 9/10 2023-11-28T14:28:47.039 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-test-2:18.0.0-7503.g5537419c.el9.x86_64 10/10 2023-11-28T14:28:50.727 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-test-2:18.0.0-7503.g5537419c.el9.x86_64 1/10 2023-11-28T14:28:50.727 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephsqlite-2:18.0.0-7503.g5537419c.el9.x86_64 2/10 2023-11-28T14:28:50.727 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2023-11-28T14:28:50.727 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 4/10 2023-11-28T14:28:50.727 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 5/10 2023-11-28T14:28:50.727 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/10 2023-11-28T14:28:50.727 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2023-11-28T14:28:50.727 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2023-11-28T14:28:50.727 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-11-28T14:28:50.729 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-11-28T14:28:50.729 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:28:50.729 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:28:50.729 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:50.729 INFO:teuthology.orchestra.run.smithi055.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-11-28T14:28:50.729 INFO:teuthology.orchestra.run.smithi055.stdout: jq-1.6-16.el9.x86_64 2023-11-28T14:28:50.729 INFO:teuthology.orchestra.run.smithi055.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-11-28T14:28:50.729 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:50.729 INFO:teuthology.orchestra.run.smithi055.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-11-28T14:28:50.729 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-28T14:28:50.730 INFO:teuthology.orchestra.run.smithi055.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-28T14:28:50.730 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-28T14:28:50.730 INFO:teuthology.orchestra.run.smithi055.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-11-28T14:28:50.730 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:28:50.730 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:28:50.730 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph 2023-11-28T14:28:51.348 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:16 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:28:51.510 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:28:51.512 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================= 2023-11-28T14:28:51.512 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-28T14:28:51.513 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================= 2023-11-28T14:28:51.513 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:28:51.513 INFO:teuthology.orchestra.run.smithi055.stdout: ceph x86_64 2:18.0.0-7503.g5537419c.el9 ceph 6.5 k 2023-11-28T14:28:51.513 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-28T14:28:51.513 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds x86_64 2:18.0.0-7503.g5537419c.el9 ceph 2.0 M 2023-11-28T14:28:51.513 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr x86_64 2:18.0.0-7503.g5537419c.el9 ceph 1.5 M 2023-11-28T14:28:51.513 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 243 k 2023-11-28T14:28:51.513 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon x86_64 2:18.0.0-7503.g5537419c.el9 ceph 8.9 M 2023-11-28T14:28:51.513 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd x86_64 2:18.0.0-7503.g5537419c.el9 ceph 118 M 2023-11-28T14:28:51.513 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-28T14:28:51.513 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-28T14:28:51.514 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-28T14:28:51.514 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-28T14:28:51.514 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-28T14:28:51.514 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-28T14:28:51.514 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-28T14:28:51.514 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-28T14:28:51.514 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-28T14:28:51.514 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-28T14:28:51.514 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-28T14:28:51.514 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-28T14:28:51.514 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-28T14:28:51.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-28T14:28:51.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-28T14:28:51.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-28T14:28:51.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-28T14:28:51.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-28T14:28:51.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-28T14:28:51.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-28T14:28:51.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-28T14:28:51.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-28T14:28:51.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-28T14:28:51.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-28T14:28:51.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-28T14:28:51.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-28T14:28:51.516 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-28T14:28:51.516 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:28:51.516 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:28:51.516 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================= 2023-11-28T14:28:51.516 INFO:teuthology.orchestra.run.smithi055.stdout:Install 33 Packages 2023-11-28T14:28:51.516 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:28:51.517 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 134 M 2023-11-28T14:28:51.517 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 1.4 G 2023-11-28T14:28:51.517 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:28:51.821 INFO:teuthology.orchestra.run.smithi055.stdout:(1/33): ceph-18.0.0-7503.g5537419c.el9.x86_64.r 29 kB/s | 6.5 kB 00:00 2023-11-28T14:28:51.971 INFO:teuthology.orchestra.run.smithi055.stdout:(2/33): ceph-mgr-18.0.0-7503.g5537419c.el9.x86_ 3.9 MB/s | 1.5 MB 00:00 2023-11-28T14:28:52.955 INFO:teuthology.orchestra.run.smithi055.stdout:(3/33): ceph-mds-18.0.0-7503.g5537419c.el9.x86_ 1.5 MB/s | 2.0 MB 00:01 2023-11-28T14:28:53.138 INFO:teuthology.orchestra.run.smithi055.stdout:(4/33): ceph-mgr-modules-core-18.0.0-7503.g5537 1.3 MB/s | 243 kB 00:00 2023-11-28T14:28:53.255 INFO:teuthology.orchestra.run.smithi055.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2023-11-28T14:28:53.280 INFO:teuthology.orchestra.run.smithi055.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-11-28T14:28:53.306 INFO:teuthology.orchestra.run.smithi055.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2023-11-28T14:28:53.331 INFO:teuthology.orchestra.run.smithi055.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2023-11-28T14:28:53.381 INFO:teuthology.orchestra.run.smithi055.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-11-28T14:28:53.456 INFO:teuthology.orchestra.run.smithi055.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2023-11-28T14:28:53.490 INFO:teuthology.orchestra.run.smithi055.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-11-28T14:28:53.523 INFO:teuthology.orchestra.run.smithi055.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2023-11-28T14:28:53.557 INFO:teuthology.orchestra.run.smithi055.stdout:(13/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-11-28T14:28:53.616 INFO:teuthology.orchestra.run.smithi055.stdout:(14/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2023-11-28T14:28:53.717 INFO:teuthology.orchestra.run.smithi055.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 436 kB/s | 43 kB 00:00 2023-11-28T14:28:53.800 INFO:teuthology.orchestra.run.smithi055.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2023-11-28T14:28:53.875 INFO:teuthology.orchestra.run.smithi055.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2023-11-28T14:28:53.917 INFO:teuthology.orchestra.run.smithi055.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2023-11-28T14:28:53.959 INFO:teuthology.orchestra.run.smithi055.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 424 kB/s | 18 kB 00:00 2023-11-28T14:28:54.001 INFO:teuthology.orchestra.run.smithi055.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 555 kB/s | 23 kB 00:00 2023-11-28T14:28:54.043 INFO:teuthology.orchestra.run.smithi055.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 468 kB/s | 19 kB 00:00 2023-11-28T14:28:54.085 INFO:teuthology.orchestra.run.smithi055.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 472 kB/s | 20 kB 00:00 2023-11-28T14:28:54.127 INFO:teuthology.orchestra.run.smithi055.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2023-11-28T14:28:54.169 INFO:teuthology.orchestra.run.smithi055.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2023-11-28T14:28:54.219 INFO:teuthology.orchestra.run.smithi055.stdout:(25/33): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2023-11-28T14:28:54.261 INFO:teuthology.orchestra.run.smithi055.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 395 kB/s | 16 kB 00:00 2023-11-28T14:28:54.303 INFO:teuthology.orchestra.run.smithi055.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2023-11-28T14:28:54.345 INFO:teuthology.orchestra.run.smithi055.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 862 kB/s | 36 kB 00:00 2023-11-28T14:28:54.396 INFO:teuthology.orchestra.run.smithi055.stdout:(29/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2023-11-28T14:28:54.454 INFO:teuthology.orchestra.run.smithi055.stdout:(30/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.2 MB/s | 427 kB 00:00 2023-11-28T14:28:54.513 INFO:teuthology.orchestra.run.smithi055.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2023-11-28T14:28:57.058 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7503.g5537419c.el9.x86_64 9/10 2023-11-28T14:28:57.132 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-2:18.0.0-7503.g5537419c.el9.x86_64 10/10 2023-11-28T14:29:01.288 INFO:teuthology.orchestra.run.smithi055.stdout:(32/33): ceph-mon-18.0.0-7503.g5537419c.el9.x86 959 kB/s | 8.9 MB 00:09 2023-11-28T14:29:05.111 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-test-2:18.0.0-7503.g5537419c.el9.x86_64 10/10 2023-11-28T14:29:05.111 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-2:18.0.0-7503.g5537419c.el9.x86_64 1/10 2023-11-28T14:29:05.111 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephsqlite-2:18.0.0-7503.g5537419c.el9.x86_64 2/10 2023-11-28T14:29:05.111 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2023-11-28T14:29:05.111 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 4/10 2023-11-28T14:29:05.111 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 5/10 2023-11-28T14:29:05.111 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/10 2023-11-28T14:29:05.112 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2023-11-28T14:29:05.112 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2023-11-28T14:29:05.112 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-11-28T14:29:05.521 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-11-28T14:29:05.522 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:29:05.522 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:29:05.522 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:29:05.522 INFO:teuthology.orchestra.run.smithi060.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-11-28T14:29:05.522 INFO:teuthology.orchestra.run.smithi060.stdout: jq-1.6-16.el9.x86_64 2023-11-28T14:29:05.522 INFO:teuthology.orchestra.run.smithi060.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-11-28T14:29:05.522 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:29:05.522 INFO:teuthology.orchestra.run.smithi060.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-11-28T14:29:05.522 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-28T14:29:05.523 INFO:teuthology.orchestra.run.smithi060.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-28T14:29:05.523 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-28T14:29:05.523 INFO:teuthology.orchestra.run.smithi060.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-11-28T14:29:05.523 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:29:05.523 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:29:05.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph 2023-11-28T14:29:06.454 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:41 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:29:06.606 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:29:06.608 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2023-11-28T14:29:06.609 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-11-28T14:29:06.609 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2023-11-28T14:29:06.609 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:29:06.609 INFO:teuthology.orchestra.run.smithi060.stdout: ceph x86_64 2:18.0.0-7503.g5537419c.el9 ceph 6.5 k 2023-11-28T14:29:06.609 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2023-11-28T14:29:06.609 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds x86_64 2:18.0.0-7503.g5537419c.el9 ceph 2.0 M 2023-11-28T14:29:06.609 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr x86_64 2:18.0.0-7503.g5537419c.el9 ceph 1.5 M 2023-11-28T14:29:06.609 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 243 k 2023-11-28T14:29:06.609 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon x86_64 2:18.0.0-7503.g5537419c.el9 ceph 8.9 M 2023-11-28T14:29:06.609 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd x86_64 2:18.0.0-7503.g5537419c.el9 ceph 118 M 2023-11-28T14:29:06.609 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-28T14:29:06.609 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-28T14:29:06.610 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-28T14:29:06.610 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-28T14:29:06.610 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-28T14:29:06.610 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-28T14:29:06.610 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-28T14:29:06.610 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-28T14:29:06.610 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-28T14:29:06.610 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-28T14:29:06.610 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-28T14:29:06.610 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-28T14:29:06.610 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-28T14:29:06.611 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-28T14:29:06.611 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-28T14:29:06.611 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-28T14:29:06.611 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-28T14:29:06.611 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-28T14:29:06.611 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-28T14:29:06.611 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-28T14:29:06.611 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-28T14:29:06.611 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-28T14:29:06.611 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-28T14:29:06.611 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-28T14:29:06.611 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-28T14:29:06.612 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-28T14:29:06.612 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-28T14:29:06.612 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:29:06.612 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:29:06.612 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2023-11-28T14:29:06.612 INFO:teuthology.orchestra.run.smithi060.stdout:Install 33 Packages 2023-11-28T14:29:06.612 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:29:06.613 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 134 M 2023-11-28T14:29:06.613 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 1.4 G 2023-11-28T14:29:06.613 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:29:07.181 INFO:teuthology.orchestra.run.smithi060.stdout:(1/33): ceph-18.0.0-7503.g5537419c.el9.x86_64.r 16 kB/s | 6.5 kB 00:00 2023-11-28T14:29:07.831 INFO:teuthology.orchestra.run.smithi060.stdout:(2/33): ceph-mgr-18.0.0-7503.g5537419c.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2023-11-28T14:29:10.831 INFO:teuthology.orchestra.run.smithi060.stdout:(3/33): ceph-mon-18.0.0-7503.g5537419c.el9.x86_ 2.4 MB/s | 8.9 MB 00:03 2023-11-28T14:29:10.998 INFO:teuthology.orchestra.run.smithi060.stdout:(4/33): ceph-mgr-modules-core-18.0.0-7503.g5537 1.4 MB/s | 243 kB 00:00 2023-11-28T14:29:11.106 INFO:teuthology.orchestra.run.smithi060.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2023-11-28T14:29:11.140 INFO:teuthology.orchestra.run.smithi060.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2023-11-28T14:29:11.174 INFO:teuthology.orchestra.run.smithi060.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-11-28T14:29:11.208 INFO:teuthology.orchestra.run.smithi060.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-11-28T14:29:11.249 INFO:teuthology.orchestra.run.smithi060.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-11-28T14:29:11.325 INFO:teuthology.orchestra.run.smithi060.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2023-11-28T14:29:11.359 INFO:teuthology.orchestra.run.smithi060.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-11-28T14:29:11.392 INFO:teuthology.orchestra.run.smithi060.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2023-11-28T14:29:11.426 INFO:teuthology.orchestra.run.smithi060.stdout:(13/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-11-28T14:29:11.459 INFO:teuthology.orchestra.run.smithi060.stdout:(14/33): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-11-28T14:29:11.709 INFO:teuthology.orchestra.run.smithi060.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 174 kB/s | 43 kB 00:00 2023-11-28T14:29:11.835 INFO:teuthology.orchestra.run.smithi060.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2023-11-28T14:29:12.018 INFO:teuthology.orchestra.run.smithi060.stdout:(17/33): ceph-mds-18.0.0-7503.g5537419c.el9.x86 390 kB/s | 2.0 MB 00:05 2023-11-28T14:29:12.077 INFO:teuthology.orchestra.run.smithi060.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2023-11-28T14:29:12.160 INFO:teuthology.orchestra.run.smithi060.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 312 kB/s | 18 kB 00:00 2023-11-28T14:29:12.219 INFO:teuthology.orchestra.run.smithi060.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 408 kB/s | 23 kB 00:00 2023-11-28T14:29:12.244 INFO:teuthology.orchestra.run.smithi060.stdout:(21/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 47 kB/s | 11 kB 00:00 2023-11-28T14:29:12.278 INFO:teuthology.orchestra.run.smithi060.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 333 kB/s | 19 kB 00:00 2023-11-28T14:29:12.303 INFO:teuthology.orchestra.run.smithi060.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 337 kB/s | 20 kB 00:00 2023-11-28T14:29:12.337 INFO:teuthology.orchestra.run.smithi060.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 789 kB/s | 46 kB 00:00 2023-11-28T14:29:12.396 INFO:teuthology.orchestra.run.smithi060.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 858 kB/s | 79 kB 00:00 2023-11-28T14:29:12.454 INFO:teuthology.orchestra.run.smithi060.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2023-11-28T14:29:12.513 INFO:teuthology.orchestra.run.smithi060.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2023-11-28T14:29:12.530 INFO:teuthology.orchestra.run.smithi060.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2023-11-28T14:29:12.572 INFO:teuthology.orchestra.run.smithi060.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 609 kB/s | 36 kB 00:00 2023-11-28T14:29:12.680 INFO:teuthology.orchestra.run.smithi060.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2023-11-28T14:29:12.739 INFO:teuthology.orchestra.run.smithi060.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2023-11-28T14:29:12.839 INFO:teuthology.orchestra.run.smithi060.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2023-11-28T14:29:26.760 INFO:teuthology.orchestra.run.smithi055.stdout:(33/33): ceph-osd-18.0.0-7503.g5537419c.el9.x86 3.4 MB/s | 118 MB 00:34 2023-11-28T14:29:26.763 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:29:26.763 INFO:teuthology.orchestra.run.smithi055.stdout:Total 3.8 MB/s | 134 MB 00:35 2023-11-28T14:29:26.763 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:29:26.800 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:29:26.801 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:29:27.812 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:29:27.812 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:29:29.716 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:29:29.771 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-11-28T14:29:29.795 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-11-28T14:29:29.828 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-11-28T14:29:29.923 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-11-28T14:29:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-11-28T14:29:30.048 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-11-28T14:29:30.122 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-11-28T14:29:30.173 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/33 2023-11-28T14:29:30.212 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2023-11-28T14:29:30.253 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2023-11-28T14:29:30.308 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2023-11-28T14:29:30.499 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2023-11-28T14:29:30.541 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2023-11-28T14:29:30.580 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2023-11-28T14:29:30.614 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2023-11-28T14:29:30.644 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2023-11-28T14:29:30.724 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2023-11-28T14:29:30.763 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-28T14:29:30.796 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2023-11-28T14:29:30.837 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2023-11-28T14:29:30.880 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2023-11-28T14:29:30.915 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-11-28T14:29:31.007 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-11-28T14:29:31.052 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-11-28T14:29:31.128 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-11-28T14:29:31.277 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-11-28T14:29:31.401 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-28T14:29:31.569 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7503.g5537419c.el9. 28/33 2023-11-28T14:29:31.600 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 29/33 2023-11-28T14:29:31.626 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 29/33 2023-11-28T14:29:31.627 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:29:31.627 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-28T14:29:31.627 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T14:29:31.627 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T14:29:31.627 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:29:38.837 INFO:teuthology.orchestra.run.smithi060.stdout:(33/33): ceph-osd-18.0.0-7503.g5537419c.el9.x86 3.8 MB/s | 118 MB 00:31 2023-11-28T14:29:38.843 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:29:38.843 INFO:teuthology.orchestra.run.smithi060.stdout:Total 4.2 MB/s | 134 MB 00:32 2023-11-28T14:29:38.843 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:29:38.886 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:29:38.886 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:29:39.895 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:29:39.895 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:29:41.900 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:29:42.089 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-11-28T14:29:42.122 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-11-28T14:29:42.155 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-11-28T14:29:42.249 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-11-28T14:29:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-11-28T14:29:42.366 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-11-28T14:29:42.438 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-11-28T14:29:42.491 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/33 2023-11-28T14:29:42.529 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2023-11-28T14:29:42.587 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2023-11-28T14:29:42.634 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2023-11-28T14:29:42.824 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2023-11-28T14:29:43.001 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2023-11-28T14:29:43.039 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2023-11-28T14:29:43.073 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2023-11-28T14:29:43.103 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2023-11-28T14:29:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2023-11-28T14:29:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-28T14:29:43.247 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2023-11-28T14:29:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2023-11-28T14:29:43.323 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2023-11-28T14:29:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-11-28T14:29:43.440 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-11-28T14:29:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-11-28T14:29:43.561 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-11-28T14:29:43.834 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-11-28T14:29:43.958 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-28T14:29:44.128 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7503.g5537419c.el9. 28/33 2023-11-28T14:29:44.159 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 29/33 2023-11-28T14:29:44.185 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 29/33 2023-11-28T14:29:44.185 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:29:44.185 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-28T14:29:44.186 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T14:29:44.186 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T14:29:44.186 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:29:51.067 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-osd-2:18.0.0-7503.g5537419c.el9.x86_64 30/33 2023-11-28T14:29:51.095 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-osd-2:18.0.0-7503.g5537419c.el9.x86_64 30/33 2023-11-28T14:29:51.096 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:29:51.096 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-28T14:29:51.096 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T14:29:51.096 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T14:29:51.096 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:29:52.026 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mon-2:18.0.0-7503.g5537419c.el9.x86_64 31/33 2023-11-28T14:29:52.050 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mon-2:18.0.0-7503.g5537419c.el9.x86_64 31/33 2023-11-28T14:29:52.050 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:29:52.050 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-28T14:29:52.050 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T14:29:52.050 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T14:29:52.050 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:29:52.268 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mds-2:18.0.0-7503.g5537419c.el9.x86_64 32/33 2023-11-28T14:29:52.292 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mds-2:18.0.0-7503.g5537419c.el9.x86_64 32/33 2023-11-28T14:29:52.292 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:29:52.292 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-28T14:29:52.292 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T14:29:52.292 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T14:29:52.292 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:29:52.360 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-2:18.0.0-7503.g5537419c.el9.x86_64 33/33 2023-11-28T14:30:02.584 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-2:18.0.0-7503.g5537419c.el9.x86_64 33/33 2023-11-28T14:30:02.584 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-2:18.0.0-7503.g5537419c.el9.x86_64 1/33 2023-11-28T14:30:02.584 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mds-2:18.0.0-7503.g5537419c.el9.x86_64 2/33 2023-11-28T14:30:02.585 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 3/33 2023-11-28T14:30:02.585 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mon-2:18.0.0-7503.g5537419c.el9.x86_64 4/33 2023-11-28T14:30:02.585 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-osd-2:18.0.0-7503.g5537419c.el9.x86_64 5/33 2023-11-28T14:30:02.585 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7503.g5537419c.el9. 6/33 2023-11-28T14:30:02.585 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-11-28T14:30:02.585 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-11-28T14:30:02.585 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2023-11-28T14:30:02.585 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2023-11-28T14:30:02.585 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2023-11-28T14:30:02.585 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2023-11-28T14:30:02.585 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2023-11-28T14:30:02.585 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2023-11-28T14:30:02.586 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2023-11-28T14:30:02.586 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 16/33 2023-11-28T14:30:02.586 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-11-28T14:30:02.586 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-28T14:30:02.586 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-11-28T14:30:02.586 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-11-28T14:30:02.586 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-11-28T14:30:02.586 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-11-28T14:30:02.586 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-11-28T14:30:02.586 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-11-28T14:30:02.586 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-11-28T14:30:02.587 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-11-28T14:30:02.587 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-28T14:30:02.588 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-11-28T14:30:02.588 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-11-28T14:30:02.588 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-11-28T14:30:02.588 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-11-28T14:30:02.588 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-11-28T14:30:03.468 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-11-28T14:30:03.468 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:03.468 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:30:03.468 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:03.468 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:03.469 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:03.469 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:03.469 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:03.469 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:03.469 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-28T14:30:03.469 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-28T14:30:03.469 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-28T14:30:03.469 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-28T14:30:03.469 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-28T14:30:03.469 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-28T14:30:03.469 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-28T14:30:03.470 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-28T14:30:03.470 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-28T14:30:03.470 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-28T14:30:03.470 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-28T14:30:03.470 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-28T14:30:03.470 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-28T14:30:03.470 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-28T14:30:03.470 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-28T14:30:03.470 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply-3.11-14.el9.noarch 2023-11-28T14:30:03.470 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-28T14:30:03.470 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-28T14:30:03.471 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-28T14:30:03.471 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-28T14:30:03.471 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-28T14:30:03.471 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-28T14:30:03.471 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-28T14:30:03.471 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-28T14:30:03.471 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-28T14:30:03.471 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-28T14:30:03.471 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-28T14:30:03.471 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:03.471 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:03.718 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-2:18.0.0-7503.g5537419c.el9.x86_64 30/33 2023-11-28T14:30:03.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-base 2023-11-28T14:30:03.749 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-2:18.0.0-7503.g5537419c.el9.x86_64 30/33 2023-11-28T14:30:03.749 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:30:03.749 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-28T14:30:03.749 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T14:30:03.750 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T14:30:03.750 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:04.334 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:29 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:04.432 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:30:04.485 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:04.486 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-28T14:30:04.486 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:04.558 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install cephadm 2023-11-28T14:30:04.685 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-2:18.0.0-7503.g5537419c.el9.x86_64 31/33 2023-11-28T14:30:04.709 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-2:18.0.0-7503.g5537419c.el9.x86_64 31/33 2023-11-28T14:30:04.709 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:30:04.709 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-28T14:30:04.709 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T14:30:04.710 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T14:30:04.710 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:04.919 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-2:18.0.0-7503.g5537419c.el9.x86_64 32/33 2023-11-28T14:30:04.943 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-2:18.0.0-7503.g5537419c.el9.x86_64 32/33 2023-11-28T14:30:04.943 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:30:04.943 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-28T14:30:04.943 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T14:30:04.943 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T14:30:04.943 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:05.011 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-2:18.0.0-7503.g5537419c.el9.x86_64 33/33 2023-11-28T14:30:05.156 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:30 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:05.306 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:05.306 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:05.306 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-28T14:30:05.307 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:05.307 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:30:05.307 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 775 k 2023-11-28T14:30:05.307 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:05.307 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:30:05.307 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:05.307 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-28T14:30:05.307 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:05.307 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 775 k 2023-11-28T14:30:05.307 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 789 k 2023-11-28T14:30:05.307 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:30:06.350 INFO:teuthology.orchestra.run.smithi055.stdout:cephadm-18.0.0-7503.g5537419c.el9.noarch.rpm 744 kB/s | 775 kB 00:01 2023-11-28T14:30:06.350 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:06.351 INFO:teuthology.orchestra.run.smithi055.stdout:Total 742 kB/s | 775 kB 00:01 2023-11-28T14:30:06.351 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:30:06.358 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:30:06.358 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:30:06.364 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:30:06.365 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:30:06.444 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:30:06.914 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:06.999 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:07.451 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:07.754 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:07.754 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:07.754 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:30:07.754 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:07.754 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:07.754 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:07.972 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-immutable-object-cache 2023-11-28T14:30:08.573 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:33 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:08.726 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:08.727 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:08.727 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2023-11-28T14:30:08.727 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:08.727 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:30:08.728 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7503.g5537419c.el9 ceph 143 k 2023-11-28T14:30:08.728 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:08.728 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:30:08.728 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:08.728 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-28T14:30:08.728 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:08.728 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 143 k 2023-11-28T14:30:08.728 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 437 k 2023-11-28T14:30:08.728 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:30:09.230 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-immutable-object-cache-18.0.0-7503.g553741 285 kB/s | 143 kB 00:00 2023-11-28T14:30:09.231 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:09.231 INFO:teuthology.orchestra.run.smithi055.stdout:Total 284 kB/s | 143 kB 00:00 2023-11-28T14:30:09.231 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:30:09.239 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:30:09.239 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:30:09.296 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:30:09.296 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:30:09.443 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:30:09.490 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7503.g5537419c. 1/1 2023-11-28T14:30:09.515 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7503.g5537419c. 1/1 2023-11-28T14:30:09.515 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:30:09.515 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-28T14:30:09.516 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:10.570 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7503.g5537419c. 1/1 2023-11-28T14:30:10.570 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:10.570 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:30:10.570 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:10.570 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:10.570 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:10.927 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr 2023-11-28T14:30:11.533 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:36 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:11.632 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:30:11.686 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:11.689 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-28T14:30:11.689 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:11.757 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-dashboard 2023-11-28T14:30:12.361 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:37 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:12.513 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:12.514 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:12.514 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-28T14:30:12.514 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:12.514 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:30:12.514 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 1.8 M 2023-11-28T14:30:12.514 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-28T14:30:12.514 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 28 k 2023-11-28T14:30:12.514 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 15 k 2023-11-28T14:30:12.514 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-28T14:30:12.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-28T14:30:12.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-28T14:30:12.515 INFO:teuthology.orchestra.run.smithi055.stdout:Installing weak dependencies: 2023-11-28T14:30:12.515 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-28T14:30:12.515 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:12.515 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:30:12.515 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:12.515 INFO:teuthology.orchestra.run.smithi055.stdout:Install 7 Packages 2023-11-28T14:30:12.515 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:12.516 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 2.1 M 2023-11-28T14:30:12.516 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 8.7 M 2023-11-28T14:30:12.516 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:30:12.960 INFO:teuthology.orchestra.run.smithi055.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7503.g553 78 kB/s | 28 kB 00:00 2023-11-28T14:30:13.001 INFO:teuthology.orchestra.run.smithi055.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7503.g5537 38 kB/s | 15 kB 00:00 2023-11-28T14:30:13.210 INFO:teuthology.orchestra.run.smithi055.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 36 kB/s | 9.0 kB 00:00 2023-11-28T14:30:13.335 INFO:teuthology.orchestra.run.smithi055.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 123 kB/s | 41 kB 00:00 2023-11-28T14:30:13.436 INFO:teuthology.orchestra.run.smithi055.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 136 kB/s | 31 kB 00:00 2023-11-28T14:30:13.602 INFO:teuthology.orchestra.run.smithi055.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 707 kB/s | 188 kB 00:00 2023-11-28T14:30:14.436 INFO:teuthology.orchestra.run.smithi055.stdout:(7/7): ceph-mgr-dashboard-18.0.0-7503.g5537419c 980 kB/s | 1.8 MB 00:01 2023-11-28T14:30:14.438 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:14.438 INFO:teuthology.orchestra.run.smithi055.stdout:Total 1.1 MB/s | 2.1 MB 00:01 2023-11-28T14:30:14.439 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:30:14.448 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:30:14.449 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:30:14.528 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:30:14.529 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:30:14.863 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:30:14.929 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-28T14:30:14.964 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-28T14:30:14.972 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-2:18.0.0-7503.g5537419c.el9.x86_64 33/33 2023-11-28T14:30:14.972 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-2:18.0.0-7503.g5537419c.el9.x86_64 1/33 2023-11-28T14:30:14.972 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-2:18.0.0-7503.g5537419c.el9.x86_64 2/33 2023-11-28T14:30:14.972 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 3/33 2023-11-28T14:30:14.972 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-2:18.0.0-7503.g5537419c.el9.x86_64 4/33 2023-11-28T14:30:14.972 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-2:18.0.0-7503.g5537419c.el9.x86_64 5/33 2023-11-28T14:30:14.972 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7503.g5537419c.el9. 6/33 2023-11-28T14:30:14.972 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-11-28T14:30:14.972 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-11-28T14:30:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2023-11-28T14:30:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2023-11-28T14:30:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2023-11-28T14:30:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2023-11-28T14:30:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2023-11-28T14:30:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2023-11-28T14:30:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2023-11-28T14:30:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 16/33 2023-11-28T14:30:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-11-28T14:30:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-28T14:30:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-11-28T14:30:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-11-28T14:30:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-11-28T14:30:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-11-28T14:30:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-11-28T14:30:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-11-28T14:30:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-11-28T14:30:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-11-28T14:30:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-28T14:30:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-11-28T14:30:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-11-28T14:30:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-11-28T14:30:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-11-28T14:30:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-11-28T14:30:15.011 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-28T14:30:15.050 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-28T14:30:15.080 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7503.g5537419c.el9.n 5/7 2023-11-28T14:30:15.328 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7503.g5537419c.el9. 6/7 2023-11-28T14:30:15.365 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7503.g5537419c.el9.noarc 7/7 2023-11-28T14:30:15.828 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-11-28T14:30:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:15.829 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:30:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-28T14:30:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-28T14:30:15.829 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-28T14:30:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-28T14:30:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-28T14:30:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-28T14:30:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-28T14:30:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-28T14:30:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-28T14:30:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-28T14:30:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-28T14:30:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-28T14:30:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-28T14:30:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-28T14:30:15.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-28T14:30:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply-3.11-14.el9.noarch 2023-11-28T14:30:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-28T14:30:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-28T14:30:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-28T14:30:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-28T14:30:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-28T14:30:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-28T14:30:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-28T14:30:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-28T14:30:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-28T14:30:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-28T14:30:15.831 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-28T14:30:15.832 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:15.832 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:30:15.917 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7503.g5537419c.el9.noarc 7/7 2023-11-28T14:30:15.917 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7503.g5537419c.el9. 1/7 2023-11-28T14:30:15.917 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7503.g5537419c.el9.noarc 2/7 2023-11-28T14:30:15.918 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7503.g5537419c.el9.n 3/7 2023-11-28T14:30:15.918 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-28T14:30:15.918 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-28T14:30:15.918 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-28T14:30:16.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-base 2023-11-28T14:30:16.247 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-28T14:30:16.247 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:16.247 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:30:16.247 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:16.247 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:16.248 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:16.248 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-28T14:30:16.248 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-28T14:30:16.248 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-28T14:30:16.248 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-28T14:30:16.248 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:16.248 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:16.455 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-28T14:30:16.688 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:51 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:30:16.786 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:30:16.838 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:30:16.839 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2023-11-28T14:30:16.840 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:30:16.900 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install cephadm 2023-11-28T14:30:17.060 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:42 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:17.212 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:17.213 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2023-11-28T14:30:17.213 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-28T14:30:17.213 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2023-11-28T14:30:17.213 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:30:17.214 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 7.4 M 2023-11-28T14:30:17.214 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-28T14:30:17.214 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-28T14:30:17.214 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-28T14:30:17.214 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-28T14:30:17.214 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-28T14:30:17.214 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-28T14:30:17.214 INFO:teuthology.orchestra.run.smithi055.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-28T14:30:17.214 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-28T14:30:17.214 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-28T14:30:17.214 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-28T14:30:17.214 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-28T14:30:17.215 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-28T14:30:17.215 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:17.215 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:30:17.215 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2023-11-28T14:30:17.215 INFO:teuthology.orchestra.run.smithi055.stdout:Install 12 Packages 2023-11-28T14:30:17.215 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:17.215 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 37 M 2023-11-28T14:30:17.216 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 209 M 2023-11-28T14:30:17.216 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:30:17.498 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:52 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:30:17.618 INFO:teuthology.orchestra.run.smithi055.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 234 kB/s | 32 kB 00:00 2023-11-28T14:30:17.649 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:30:17.650 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:17.650 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-11-28T14:30:17.650 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:17.650 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:30:17.650 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 775 k 2023-11-28T14:30:17.650 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:17.650 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:30:17.650 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:17.650 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2023-11-28T14:30:17.650 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:17.650 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 775 k 2023-11-28T14:30:17.651 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 789 k 2023-11-28T14:30:17.651 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:30:17.651 INFO:teuthology.orchestra.run.smithi055.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 515 kB/s | 17 kB 00:00 2023-11-28T14:30:17.693 INFO:teuthology.orchestra.run.smithi055.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 852 kB/s | 34 kB 00:00 2023-11-28T14:30:17.752 INFO:teuthology.orchestra.run.smithi055.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-11-28T14:30:17.894 INFO:teuthology.orchestra.run.smithi055.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2023-11-28T14:30:17.961 INFO:teuthology.orchestra.run.smithi055.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2023-11-28T14:30:17.986 INFO:teuthology.orchestra.run.smithi055.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2023-11-28T14:30:18.303 INFO:teuthology.orchestra.run.smithi055.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2023-11-28T14:30:18.328 INFO:teuthology.orchestra.run.smithi055.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2023-11-28T14:30:18.353 INFO:teuthology.orchestra.run.smithi055.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.9 MB/s | 195 kB 00:00 2023-11-28T14:30:18.521 INFO:teuthology.orchestra.run.smithi055.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2023-11-28T14:30:18.534 INFO:teuthology.orchestra.run.smithi060.stdout:cephadm-18.0.0-7503.g5537419c.el9.noarch.rpm 878 kB/s | 775 kB 00:00 2023-11-28T14:30:18.534 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:18.535 INFO:teuthology.orchestra.run.smithi060.stdout:Total 876 kB/s | 775 kB 00:00 2023-11-28T14:30:18.535 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:30:18.542 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:30:18.542 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:30:18.548 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:30:18.549 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:30:18.626 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:30:19.073 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:19.157 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:19.638 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:19.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:19.951 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:19.951 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:30:19.952 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:19.952 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:19.952 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:30:20.142 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-immutable-object-cache 2023-11-28T14:30:20.748 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:55 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:30:20.900 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:30:20.901 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:20.901 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2023-11-28T14:30:20.901 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:20.901 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:30:20.901 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7503.g5537419c.el9 ceph 143 k 2023-11-28T14:30:20.901 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:20.901 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:30:20.901 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:20.901 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2023-11-28T14:30:20.901 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:20.902 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 143 k 2023-11-28T14:30:20.902 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 437 k 2023-11-28T14:30:20.902 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:30:21.873 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-immutable-object-cache-18.0.0-7503.g553741 148 kB/s | 143 kB 00:00 2023-11-28T14:30:21.873 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:21.874 INFO:teuthology.orchestra.run.smithi060.stdout:Total 147 kB/s | 143 kB 00:00 2023-11-28T14:30:21.874 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:30:21.884 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:30:21.884 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:30:21.942 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:30:21.943 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:30:22.086 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:30:22.232 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7503.g5537419c. 1/1 2023-11-28T14:30:22.258 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7503.g5537419c. 1/1 2023-11-28T14:30:22.258 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:30:22.259 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-28T14:30:22.259 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 1.3 MB/s | 7.4 MB 00:05 2023-11-28T14:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout:Total 6.2 MB/s | 37 MB 00:05 2023-11-28T14:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:30:23.231 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7503.g5537419c. 1/1 2023-11-28T14:30:23.231 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:23.231 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:30:23.231 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:23.231 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:23.231 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:30:23.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr 2023-11-28T14:30:23.670 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:30:23.673 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:30:24.078 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:59 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:30:24.177 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:30:24.230 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:30:24.231 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2023-11-28T14:30:24.232 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:30:24.302 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-dashboard 2023-11-28T14:30:24.430 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:30:24.502 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-28T14:30:24.533 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-28T14:30:24.633 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-28T14:30:24.670 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-28T14:30:24.869 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-28T14:30:24.902 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:59 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:30:24.902 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-28T14:30:24.990 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-28T14:30:25.054 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:30:25.055 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-11-28T14:30:25.056 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:25.056 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:30:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 1.8 M 2023-11-28T14:30:25.056 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2023-11-28T14:30:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 28 k 2023-11-28T14:30:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 15 k 2023-11-28T14:30:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-28T14:30:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-28T14:30:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-28T14:30:25.056 INFO:teuthology.orchestra.run.smithi060.stdout:Installing weak dependencies: 2023-11-28T14:30:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-28T14:30:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:25.057 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:30:25.057 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:25.057 INFO:teuthology.orchestra.run.smithi060.stdout:Install 7 Packages 2023-11-28T14:30:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:25.057 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 2.1 M 2023-11-28T14:30:25.057 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 8.7 M 2023-11-28T14:30:25.057 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:30:25.643 INFO:teuthology.orchestra.run.smithi060.stdout:(1/7): ceph-prometheus-alerts-18.0.0-7503.g5537 38 kB/s | 15 kB 00:00 2023-11-28T14:30:25.659 INFO:teuthology.orchestra.run.smithi060.stdout:(2/7): ceph-grafana-dashboards-18.0.0-7503.g553 67 kB/s | 28 kB 00:00 2023-11-28T14:30:25.694 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-28T14:30:25.777 INFO:teuthology.orchestra.run.smithi060.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 68 kB/s | 9.0 kB 00:00 2023-11-28T14:30:25.810 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-28T14:30:25.868 INFO:teuthology.orchestra.run.smithi060.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 196 kB/s | 41 kB 00:00 2023-11-28T14:30:25.885 INFO:teuthology.orchestra.run.smithi060.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 283 kB/s | 31 kB 00:00 2023-11-28T14:30:26.027 INFO:teuthology.orchestra.run.smithi060.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2023-11-28T14:30:26.519 INFO:teuthology.orchestra.run.smithi060.stdout:(7/7): ceph-mgr-dashboard-18.0.0-7503.g5537419c 1.4 MB/s | 1.8 MB 00:01 2023-11-28T14:30:26.521 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:26.521 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.4 MB/s | 2.1 MB 00:01 2023-11-28T14:30:26.522 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:30:26.531 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:30:26.532 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:30:26.609 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:30:26.609 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:30:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:30:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-28T14:30:27.005 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-28T14:30:27.053 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-28T14:30:27.083 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-28T14:30:27.112 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-28T14:30:27.113 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7503.g5537419c.el9.n 5/7 2023-11-28T14:30:27.363 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7503.g5537419c.el9. 6/7 2023-11-28T14:30:27.398 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7503.g5537419c.el9.noarc 7/7 2023-11-28T14:30:27.946 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7503.g5537419c.el9.noarc 7/7 2023-11-28T14:30:27.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7503.g5537419c.el9. 1/7 2023-11-28T14:30:27.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7503.g5537419c.el9.noarc 2/7 2023-11-28T14:30:27.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7503.g5537419c.el9.n 3/7 2023-11-28T14:30:27.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-28T14:30:27.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-28T14:30:27.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-28T14:30:28.205 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-28T14:30:28.236 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7503.g55374 12/12 2023-11-28T14:30:28.263 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-28T14:30:28.263 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:28.264 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:30:28.264 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:28.264 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:28.264 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:28.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-28T14:30:28.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-28T14:30:28.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-28T14:30:28.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-28T14:30:28.264 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:28.264 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:30:28.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-28T14:30:29.105 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:04 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:30:29.259 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:30:29.261 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2023-11-28T14:30:29.261 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-11-28T14:30:29.261 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2023-11-28T14:30:29.261 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:30:29.261 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 7.4 M 2023-11-28T14:30:29.261 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2023-11-28T14:30:29.262 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-28T14:30:29.262 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-28T14:30:29.262 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-28T14:30:29.262 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-28T14:30:29.262 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-28T14:30:29.262 INFO:teuthology.orchestra.run.smithi060.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-28T14:30:29.262 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-28T14:30:29.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-28T14:30:29.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-28T14:30:29.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-28T14:30:29.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-28T14:30:29.263 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:29.263 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:30:29.263 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2023-11-28T14:30:29.263 INFO:teuthology.orchestra.run.smithi060.stdout:Install 12 Packages 2023-11-28T14:30:29.263 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:29.264 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 37 M 2023-11-28T14:30:29.264 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 209 M 2023-11-28T14:30:29.264 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:30:29.443 INFO:teuthology.orchestra.run.smithi060.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 233 kB/s | 32 kB 00:00 2023-11-28T14:30:29.476 INFO:teuthology.orchestra.run.smithi060.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 515 kB/s | 17 kB 00:00 2023-11-28T14:30:29.518 INFO:teuthology.orchestra.run.smithi060.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 825 kB/s | 34 kB 00:00 2023-11-28T14:30:29.585 INFO:teuthology.orchestra.run.smithi060.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-11-28T14:30:29.727 INFO:teuthology.orchestra.run.smithi060.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2023-11-28T14:30:29.811 INFO:teuthology.orchestra.run.smithi060.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2023-11-28T14:30:29.836 INFO:teuthology.orchestra.run.smithi060.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2023-11-28T14:30:30.069 INFO:teuthology.orchestra.run.smithi060.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2023-11-28T14:30:30.253 INFO:teuthology.orchestra.run.smithi060.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2023-11-28T14:30:30.278 INFO:teuthology.orchestra.run.smithi060.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 947 kB/s | 195 kB 00:00 2023-11-28T14:30:30.478 INFO:teuthology.orchestra.run.smithi060.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2023-11-28T14:30:30.801 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7503.g55374 12/12 2023-11-28T14:30:30.801 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7503.g55374 1/12 2023-11-28T14:30:30.801 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-28T14:30:30.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-28T14:30:30.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-28T14:30:30.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-28T14:30:30.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-28T14:30:30.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-11-28T14:30:30.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-11-28T14:30:30.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-11-28T14:30:30.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-11-28T14:30:30.802 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-11-28T14:30:31.248 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-11-28T14:30:31.248 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:31.248 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:30:31.248 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:31.248 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-28T14:30:31.248 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-28T14:30:31.248 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-28T14:30:31.248 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-28T14:30:31.248 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-28T14:30:31.249 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-28T14:30:31.249 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-28T14:30:31.249 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-28T14:30:31.249 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-28T14:30:31.249 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-28T14:30:31.249 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-28T14:30:31.249 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:31.249 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:31.583 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-rook 2023-11-28T14:30:32.180 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:57 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:32.332 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:32.333 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2023-11-28T14:30:32.333 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-28T14:30:32.333 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2023-11-28T14:30:32.333 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:30:32.333 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 50 k 2023-11-28T14:30:32.334 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-28T14:30:32.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-28T14:30:32.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-28T14:30:32.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-28T14:30:32.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-28T14:30:32.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-28T14:30:32.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-28T14:30:32.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-28T14:30:32.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-28T14:30:32.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-28T14:30:32.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-28T14:30:32.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-28T14:30:32.335 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-28T14:30:32.335 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:32.335 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:30:32.335 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2023-11-28T14:30:32.335 INFO:teuthology.orchestra.run.smithi055.stdout:Install 13 Packages 2023-11-28T14:30:32.335 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:32.335 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 2.0 M 2023-11-28T14:30:32.336 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 26 M 2023-11-28T14:30:32.336 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:30:32.362 INFO:teuthology.orchestra.run.smithi060.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 2.4 MB/s | 7.4 MB 00:03 2023-11-28T14:30:32.365 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:32.366 INFO:teuthology.orchestra.run.smithi060.stdout:Total 12 MB/s | 37 MB 00:03 2023-11-28T14:30:32.381 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:30:32.409 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:30:32.409 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:30:32.628 INFO:teuthology.orchestra.run.smithi055.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 145 kB/s | 18 kB 00:00 2023-11-28T14:30:32.653 INFO:teuthology.orchestra.run.smithi055.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 163 kB/s | 24 kB 00:00 2023-11-28T14:30:32.704 INFO:teuthology.orchestra.run.smithi055.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-11-28T14:30:32.729 INFO:teuthology.orchestra.run.smithi055.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-11-28T14:30:32.754 INFO:teuthology.orchestra.run.smithi055.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-11-28T14:30:32.780 INFO:teuthology.orchestra.run.smithi055.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2023-11-28T14:30:32.856 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:30:32.857 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:30:32.888 INFO:teuthology.orchestra.run.smithi055.stdout:(7/13): ceph-mgr-rook-18.0.0-7503.g5537419c.el9 130 kB/s | 50 kB 00:00 2023-11-28T14:30:33.205 INFO:teuthology.orchestra.run.smithi055.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 35 kB/s | 15 kB 00:00 2023-11-28T14:30:33.314 INFO:teuthology.orchestra.run.smithi055.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 58 kB/s | 32 kB 00:00 2023-11-28T14:30:33.564 INFO:teuthology.orchestra.run.smithi055.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 236 kB/s | 59 kB 00:00 2023-11-28T14:30:33.593 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:30:33.656 INFO:teuthology.orchestra.run.smithi055.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 275 kB/s | 197 kB 00:00 2023-11-28T14:30:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-28T14:30:33.699 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-28T14:30:33.789 INFO:teuthology.orchestra.run.smithi055.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 398 kB/s | 90 kB 00:00 2023-11-28T14:30:33.793 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-28T14:30:33.828 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-28T14:30:33.873 INFO:teuthology.orchestra.run.smithi055.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2023-11-28T14:30:33.878 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:33.878 INFO:teuthology.orchestra.run.smithi055.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2023-11-28T14:30:33.879 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:30:33.895 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:30:33.895 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:30:34.010 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:30:34.011 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:30:34.022 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-28T14:30:34.052 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-28T14:30:34.158 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-28T14:30:34.539 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:30:34.712 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-28T14:30:34.767 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-28T14:30:34.824 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-28T14:30:34.860 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-28T14:30:34.867 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-28T14:30:34.909 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-28T14:30:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-28T14:30:35.006 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-28T14:30:35.126 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-11-28T14:30:35.178 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-28T14:30:35.946 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-28T14:30:36.007 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-28T14:30:36.041 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-28T14:30:36.090 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-28T14:30:36.119 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-rook-2:18.0.0-7503.g5537419c.el9.noarch 13/13 2023-11-28T14:30:36.301 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-28T14:30:36.896 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7503.g5537419c.el9.noarch 13/13 2023-11-28T14:30:36.896 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7503.g5537419c.el9.noarch 1/13 2023-11-28T14:30:36.896 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-28T14:30:36.896 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-28T14:30:36.896 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-28T14:30:36.896 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-28T14:30:36.896 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-28T14:30:36.896 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-28T14:30:36.896 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-28T14:30:36.897 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-28T14:30:36.897 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-11-28T14:30:36.897 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-28T14:30:36.897 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-28T14:30:37.278 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-28T14:30:37.278 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:37.278 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:30:37.278 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:37.278 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-28T14:30:37.278 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-28T14:30:37.279 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-28T14:30:37.279 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-28T14:30:37.279 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-28T14:30:37.279 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-28T14:30:37.279 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-28T14:30:37.279 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-28T14:30:37.279 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-28T14:30:37.279 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-28T14:30:37.280 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-28T14:30:37.280 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-28T14:30:37.280 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:37.280 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:37.404 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-28T14:30:37.434 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7503.g55374 12/12 2023-11-28T14:30:37.541 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-cephadm 2023-11-28T14:30:38.163 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:03 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:38.315 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:38.316 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2023-11-28T14:30:38.316 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-28T14:30:38.316 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2023-11-28T14:30:38.316 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:30:38.316 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 133 k 2023-11-28T14:30:38.316 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-28T14:30:38.316 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-28T14:30:38.316 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-28T14:30:38.317 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-28T14:30:38.317 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-28T14:30:38.317 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-28T14:30:38.317 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:38.317 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:30:38.317 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2023-11-28T14:30:38.317 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2023-11-28T14:30:38.317 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:38.318 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 6.8 M 2023-11-28T14:30:38.318 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 33 M 2023-11-28T14:30:38.318 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:30:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-11-28T14:30:38.801 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2023-11-28T14:30:38.884 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 696 kB/s | 58 kB 00:00 2023-11-28T14:30:38.918 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.0 MB/s | 548 kB 00:00 2023-11-28T14:30:38.935 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2023-11-28T14:30:39.110 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): ceph-mgr-cephadm-18.0.0-7503.g5537419c.e 211 kB/s | 133 kB 00:00 2023-11-28T14:30:39.112 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:39.112 INFO:teuthology.orchestra.run.smithi055.stdout:Total 8.5 MB/s | 6.8 MB 00:00 2023-11-28T14:30:39.112 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:30:39.123 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:30:39.123 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:30:39.218 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:30:39.219 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:30:39.520 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:30:39.682 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-28T14:30:39.738 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-28T14:30:39.871 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7503.g55374 12/12 2023-11-28T14:30:39.871 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7503.g55374 1/12 2023-11-28T14:30:39.872 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-28T14:30:39.872 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-28T14:30:39.872 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-28T14:30:39.872 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-28T14:30:39.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-28T14:30:39.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-11-28T14:30:39.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-11-28T14:30:39.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-11-28T14:30:39.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-11-28T14:30:39.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-11-28T14:30:40.289 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-28T14:30:40.374 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-28T14:30:40.375 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-11-28T14:30:40.375 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:40.375 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:30:40.375 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:40.376 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-28T14:30:40.376 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-28T14:30:40.376 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-28T14:30:40.376 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-28T14:30:40.376 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-28T14:30:40.376 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-28T14:30:40.376 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-28T14:30:40.376 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-28T14:30:40.376 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-28T14:30:40.376 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-28T14:30:40.376 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-28T14:30:40.377 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:40.377 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:30:40.435 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-28T14:30:40.458 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7503.g5537419c.el9.noarch 6/6 2023-11-28T14:30:40.740 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-rook 2023-11-28T14:30:41.273 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7503.g5537419c.el9.noarch 6/6 2023-11-28T14:30:41.273 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/6 2023-11-28T14:30:41.274 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-28T14:30:41.274 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-28T14:30:41.274 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-28T14:30:41.274 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-28T14:30:41.345 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:16 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:30:41.497 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:30:41.499 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2023-11-28T14:30:41.499 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-11-28T14:30:41.499 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2023-11-28T14:30:41.499 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:30:41.499 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 50 k 2023-11-28T14:30:41.499 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2023-11-28T14:30:41.499 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-28T14:30:41.499 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-28T14:30:41.499 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-28T14:30:41.499 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-28T14:30:41.500 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-28T14:30:41.500 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-28T14:30:41.500 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-28T14:30:41.500 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-28T14:30:41.500 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-28T14:30:41.500 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-28T14:30:41.500 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-28T14:30:41.500 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-28T14:30:41.500 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:41.500 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:30:41.500 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2023-11-28T14:30:41.501 INFO:teuthology.orchestra.run.smithi060.stdout:Install 13 Packages 2023-11-28T14:30:41.501 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:41.501 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 2.0 M 2023-11-28T14:30:41.501 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 26 M 2023-11-28T14:30:41.501 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:30:41.592 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-28T14:30:41.593 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:41.593 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:30:41.593 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:41.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-28T14:30:41.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-28T14:30:41.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-28T14:30:41.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-28T14:30:41.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-28T14:30:41.593 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:41.593 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:41.787 INFO:teuthology.orchestra.run.smithi060.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 190 kB/s | 24 kB 00:00 2023-11-28T14:30:41.812 INFO:teuthology.orchestra.run.smithi060.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2023-11-28T14:30:41.859 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-fuse 2023-11-28T14:30:41.862 INFO:teuthology.orchestra.run.smithi060.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-11-28T14:30:41.888 INFO:teuthology.orchestra.run.smithi060.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-11-28T14:30:41.913 INFO:teuthology.orchestra.run.smithi060.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-11-28T14:30:41.938 INFO:teuthology.orchestra.run.smithi060.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2023-11-28T14:30:42.030 INFO:teuthology.orchestra.run.smithi060.stdout:(7/13): ceph-mgr-rook-18.0.0-7503.g5537419c.el9 135 kB/s | 50 kB 00:00 2023-11-28T14:30:42.114 INFO:teuthology.orchestra.run.smithi060.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 85 kB/s | 15 kB 00:00 2023-11-28T14:30:42.139 INFO:teuthology.orchestra.run.smithi060.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 143 kB/s | 32 kB 00:00 2023-11-28T14:30:42.198 INFO:teuthology.orchestra.run.smithi060.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2023-11-28T14:30:42.273 INFO:teuthology.orchestra.run.smithi060.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 823 kB/s | 197 kB 00:00 2023-11-28T14:30:42.298 INFO:teuthology.orchestra.run.smithi060.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 895 kB/s | 90 kB 00:00 2023-11-28T14:30:42.464 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:07 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:42.515 INFO:teuthology.orchestra.run.smithi060.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.6 MB/s | 1.0 MB 00:00 2023-11-28T14:30:42.519 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:42.519 INFO:teuthology.orchestra.run.smithi060.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2023-11-28T14:30:42.520 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:30:42.534 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:30:42.534 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:30:42.613 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:42.614 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:42.614 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-28T14:30:42.614 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:42.614 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:30:42.614 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse x86_64 2:18.0.0-7503.g5537419c.el9 ceph 832 k 2023-11-28T14:30:42.614 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-28T14:30:42.614 INFO:teuthology.orchestra.run.smithi055.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-28T14:30:42.614 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:42.614 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:30:42.614 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:42.615 INFO:teuthology.orchestra.run.smithi055.stdout:Install 2 Packages 2023-11-28T14:30:42.615 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:42.615 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 911 k 2023-11-28T14:30:42.615 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 3.0 M 2023-11-28T14:30:42.615 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:30:42.640 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:30:42.642 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:30:42.782 INFO:teuthology.orchestra.run.smithi055.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 471 kB/s | 79 kB 00:00 2023-11-28T14:30:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:30:43.387 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-28T14:30:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-28T14:30:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-28T14:30:43.534 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-28T14:30:43.576 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-28T14:30:43.604 INFO:teuthology.orchestra.run.smithi055.stdout:(2/2): ceph-fuse-18.0.0-7503.g5537419c.el9.x86_ 842 kB/s | 832 kB 00:00 2023-11-28T14:30:43.606 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:43.606 INFO:teuthology.orchestra.run.smithi055.stdout:Total 919 kB/s | 911 kB 00:00 2023-11-28T14:30:43.606 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:30:43.631 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:30:43.631 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:30:43.675 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-28T14:30:43.714 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:30:43.714 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:30:43.785 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-11-28T14:30:43.837 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-28T14:30:43.909 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:30:44.066 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-28T14:30:44.139 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 2/2 2023-11-28T14:30:44.614 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-28T14:30:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-28T14:30:44.708 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-28T14:30:44.757 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-28T14:30:44.785 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 2/2 2023-11-28T14:30:44.785 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 1/2 2023-11-28T14:30:44.786 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-rook-2:18.0.0-7503.g5537419c.el9.noarch 13/13 2023-11-28T14:30:45.091 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-28T14:30:45.091 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:45.091 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:30:45.091 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-28T14:30:45.091 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:45.091 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:45.375 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-volume 2023-11-28T14:30:45.602 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7503.g5537419c.el9.noarch 13/13 2023-11-28T14:30:45.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7503.g5537419c.el9.noarch 1/13 2023-11-28T14:30:45.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-28T14:30:45.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-28T14:30:45.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-28T14:30:45.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-28T14:30:45.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-28T14:30:45.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-28T14:30:45.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-28T14:30:45.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-28T14:30:45.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-11-28T14:30:45.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-28T14:30:45.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-28T14:30:45.978 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:10 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:45.980 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-28T14:30:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:45.981 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:30:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-28T14:30:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-28T14:30:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-28T14:30:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-28T14:30:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-28T14:30:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-28T14:30:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-28T14:30:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-28T14:30:45.982 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-28T14:30:45.982 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-28T14:30:45.982 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-28T14:30:45.982 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-28T14:30:45.982 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:45.982 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:30:46.130 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:46.131 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:46.131 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-28T14:30:46.131 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:46.131 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:30:46.131 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 262 k 2023-11-28T14:30:46.132 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:46.132 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:30:46.132 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:46.132 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-28T14:30:46.132 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:46.132 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 262 k 2023-11-28T14:30:46.132 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 1.2 M 2023-11-28T14:30:46.133 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:30:46.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-cephadm 2023-11-28T14:30:46.517 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-volume-18.0.0-7503.g5537419c.el9.noarch.rp 682 kB/s | 262 kB 00:00 2023-11-28T14:30:46.518 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:46.518 INFO:teuthology.orchestra.run.smithi055.stdout:Total 679 kB/s | 262 kB 00:00 2023-11-28T14:30:46.518 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:30:46.527 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:30:46.527 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:30:46.569 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:30:46.569 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:30:46.795 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:30:46.853 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:21 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:30:46.886 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-volume-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:46.902 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-volume-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:46.902 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:30:46.902 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-28T14:30:46.903 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:47.005 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:30:47.006 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2023-11-28T14:30:47.006 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-11-28T14:30:47.006 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2023-11-28T14:30:47.006 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:30:47.006 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 133 k 2023-11-28T14:30:47.006 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2023-11-28T14:30:47.006 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-28T14:30:47.007 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-28T14:30:47.007 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-28T14:30:47.007 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-28T14:30:47.007 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-28T14:30:47.007 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:47.007 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:30:47.007 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2023-11-28T14:30:47.007 INFO:teuthology.orchestra.run.smithi060.stdout:Install 6 Packages 2023-11-28T14:30:47.007 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:47.008 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 6.8 M 2023-11-28T14:30:47.008 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 33 M 2023-11-28T14:30:47.008 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:30:47.351 INFO:teuthology.orchestra.run.smithi060.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-11-28T14:30:47.534 INFO:teuthology.orchestra.run.smithi060.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2023-11-28T14:30:47.635 INFO:teuthology.orchestra.run.smithi060.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 578 kB/s | 58 kB 00:00 2023-11-28T14:30:47.660 INFO:teuthology.orchestra.run.smithi060.stdout:(4/6): ceph-mgr-cephadm-18.0.0-7503.g5537419c.e 275 kB/s | 133 kB 00:00 2023-11-28T14:30:47.685 INFO:teuthology.orchestra.run.smithi060.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2023-11-28T14:30:47.710 INFO:teuthology.orchestra.run.smithi060.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2023-11-28T14:30:47.712 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:47.712 INFO:teuthology.orchestra.run.smithi060.stdout:Total 9.6 MB/s | 6.8 MB 00:00 2023-11-28T14:30:47.713 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:30:47.722 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:30:47.722 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:30:47.816 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:30:47.817 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:30:48.013 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-volume-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:48.013 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:48.013 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:30:48.013 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:48.014 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:48.014 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:48.118 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:30:48.295 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-28T14:30:48.333 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados-devel 2023-11-28T14:30:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-28T14:30:48.919 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-28T14:30:48.942 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:13 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:49.001 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-28T14:30:49.069 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-28T14:30:49.096 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:49.097 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:49.097 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2023-11-28T14:30:49.097 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:49.098 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:30:49.098 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel x86_64 2:18.0.0-7503.g5537419c.el9 ceph 127 k 2023-11-28T14:30:49.098 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:49.098 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:30:49.098 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:49.098 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-28T14:30:49.098 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:49.098 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 127 k 2023-11-28T14:30:49.098 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 456 k 2023-11-28T14:30:49.098 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:30:49.099 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7503.g5537419c.el9.noarch 6/6 2023-11-28T14:30:49.448 INFO:teuthology.orchestra.run.smithi055.stdout:librados-devel-18.0.0-7503.g5537419c.el9.x86_64 362 kB/s | 127 kB 00:00 2023-11-28T14:30:49.449 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:49.449 INFO:teuthology.orchestra.run.smithi055.stdout:Total 361 kB/s | 127 kB 00:00 2023-11-28T14:30:49.449 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:30:49.455 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:30:49.455 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:30:49.514 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:30:49.515 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:30:49.670 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:30:49.773 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librados-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:30:49.943 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7503.g5537419c.el9.noarch 6/6 2023-11-28T14:30:49.943 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/6 2023-11-28T14:30:49.943 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-28T14:30:49.943 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-28T14:30:49.943 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-28T14:30:49.943 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-28T14:30:50.211 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:30:50.253 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-28T14:30:50.253 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:50.253 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:30:50.253 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:50.253 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-28T14:30:50.253 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-28T14:30:50.254 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-28T14:30:50.254 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-28T14:30:50.254 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-28T14:30:50.254 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:50.254 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:30:50.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-fuse 2023-11-28T14:30:50.551 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:30:50.551 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:50.551 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:30:50.551 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:50.551 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:50.551 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:50.793 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs2 2023-11-28T14:30:51.135 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:26 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:30:51.285 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:30:51.286 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-11-28T14:30:51.286 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:51.286 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:30:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse x86_64 2:18.0.0-7503.g5537419c.el9 ceph 832 k 2023-11-28T14:30:51.286 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2023-11-28T14:30:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-28T14:30:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:51.287 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:30:51.287 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:51.287 INFO:teuthology.orchestra.run.smithi060.stdout:Install 2 Packages 2023-11-28T14:30:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:51.287 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 911 k 2023-11-28T14:30:51.288 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 3.0 M 2023-11-28T14:30:51.288 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:30:51.393 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:16 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:51.421 INFO:teuthology.orchestra.run.smithi060.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 592 kB/s | 79 kB 00:00 2023-11-28T14:30:51.491 INFO:teuthology.orchestra.run.smithi055.stdout:Package libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:30:51.546 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:51.547 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-28T14:30:51.547 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:51.618 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs-devel 2023-11-28T14:30:52.218 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:17 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:52.237 INFO:teuthology.orchestra.run.smithi060.stdout:(2/2): ceph-fuse-18.0.0-7503.g5537419c.el9.x86_ 877 kB/s | 832 kB 00:00 2023-11-28T14:30:52.239 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:52.239 INFO:teuthology.orchestra.run.smithi060.stdout:Total 956 kB/s | 911 kB 00:00 2023-11-28T14:30:52.240 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:30:52.264 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:30:52.264 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:30:52.349 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:30:52.349 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:30:52.370 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:52.371 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:52.371 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2023-11-28T14:30:52.371 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:52.371 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:30:52.371 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel x86_64 2:18.0.0-7503.g5537419c.el9 ceph 31 k 2023-11-28T14:30:52.371 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:52.371 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:30:52.371 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:52.371 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-28T14:30:52.371 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:52.372 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 31 k 2023-11-28T14:30:52.372 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 139 k 2023-11-28T14:30:52.372 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:30:52.547 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:30:52.619 INFO:teuthology.orchestra.run.smithi055.stdout:libcephfs-devel-18.0.0-7503.g5537419c.el9.x86_6 126 kB/s | 31 kB 00:00 2023-11-28T14:30:52.619 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:52.620 INFO:teuthology.orchestra.run.smithi055.stdout:Total 125 kB/s | 31 kB 00:00 2023-11-28T14:30:52.620 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:30:52.623 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:30:52.623 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:30:52.647 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:30:52.647 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:30:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-28T14:30:52.731 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:30:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 2/2 2023-11-28T14:30:52.852 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:30:53.255 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:30:53.428 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 2/2 2023-11-28T14:30:53.428 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 1/2 2023-11-28T14:30:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:30:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:53.549 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:30:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:53.549 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:53.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados2 2023-11-28T14:30:53.770 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-28T14:30:53.770 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:53.770 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:30:53.770 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-28T14:30:53.770 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:53.770 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:30:54.064 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-volume 2023-11-28T14:30:54.321 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:19 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:54.417 INFO:teuthology.orchestra.run.smithi055.stdout:Package librados2-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:30:54.470 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:54.472 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-28T14:30:54.472 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:54.543 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librbd1 2023-11-28T14:30:54.665 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:29 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:30:54.816 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:30:54.816 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-11-28T14:30:54.817 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:54.817 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:30:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 262 k 2023-11-28T14:30:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:54.817 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:30:54.817 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:54.817 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2023-11-28T14:30:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:54.817 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 262 k 2023-11-28T14:30:54.817 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 1.2 M 2023-11-28T14:30:54.818 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:30:55.145 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:20 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:55.241 INFO:teuthology.orchestra.run.smithi055.stdout:Package librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:30:55.275 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-volume-18.0.0-7503.g5537419c.el9.noarch.rp 572 kB/s | 262 kB 00:00 2023-11-28T14:30:55.276 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:55.276 INFO:teuthology.orchestra.run.smithi060.stdout:Total 571 kB/s | 262 kB 00:00 2023-11-28T14:30:55.277 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:30:55.286 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:30:55.286 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:30:55.294 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:55.295 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-28T14:30:55.296 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:55.327 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:30:55.327 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:30:55.367 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rados 2023-11-28T14:30:55.553 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:30:55.627 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-volume-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:55.644 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-volume-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:55.645 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:30:55.645 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-28T14:30:55.645 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:55.963 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:20 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:56.063 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rados-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:30:56.118 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:56.119 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-28T14:30:56.119 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:56.188 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rgw 2023-11-28T14:30:56.709 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-volume-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:30:56.710 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:56.710 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:30:56.710 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:30:56.710 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:56.710 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:30:56.784 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:21 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:56.883 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rgw-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:30:56.937 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:56.938 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-28T14:30:56.938 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:57.010 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-cephfs 2023-11-28T14:30:57.049 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados-devel 2023-11-28T14:30:57.614 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:22 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:57.664 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:32 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:30:57.714 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-cephfs-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:30:57.767 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:57.769 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-28T14:30:57.769 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:57.817 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:30:57.818 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:57.818 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2023-11-28T14:30:57.818 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:57.818 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:30:57.818 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel x86_64 2:18.0.0-7503.g5537419c.el9 ceph 127 k 2023-11-28T14:30:57.818 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:57.818 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:30:57.818 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:30:57.818 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2023-11-28T14:30:57.819 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:57.819 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 127 k 2023-11-28T14:30:57.819 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 456 k 2023-11-28T14:30:57.819 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:30:57.838 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rbd 2023-11-28T14:30:58.348 INFO:teuthology.orchestra.run.smithi060.stdout:librados-devel-18.0.0-7503.g5537419c.el9.x86_64 240 kB/s | 127 kB 00:00 2023-11-28T14:30:58.348 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:58.349 INFO:teuthology.orchestra.run.smithi060.stdout:Total 239 kB/s | 127 kB 00:00 2023-11-28T14:30:58.349 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:30:58.355 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:30:58.355 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:30:58.413 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:30:58.413 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:30:58.435 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:23 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:58.534 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rbd-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:30:58.560 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:30:58.588 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:58.590 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-28T14:30:58.590 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:30:58.659 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-fuse 2023-11-28T14:30:58.666 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:30:59.134 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:30:59.262 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:24 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:30:59.411 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:30:59.411 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:59.412 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-11-28T14:30:59.412 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:59.412 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:30:59.412 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse x86_64 2:18.0.0-7503.g5537419c.el9 ceph 86 k 2023-11-28T14:30:59.412 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:59.412 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:30:59.412 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:30:59.412 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-28T14:30:59.412 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:30:59.413 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 86 k 2023-11-28T14:30:59.413 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 256 k 2023-11-28T14:30:59.413 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:30:59.457 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:30:59.457 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:59.458 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:30:59.458 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:59.458 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:30:59.458 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:30:59.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs2 2023-11-28T14:30:59.797 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-fuse-18.0.0-7503.g5537419c.el9.x86_64.rpm 223 kB/s | 86 kB 00:00 2023-11-28T14:30:59.797 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:59.797 INFO:teuthology.orchestra.run.smithi055.stdout:Total 223 kB/s | 86 kB 00:00 2023-11-28T14:30:59.798 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:30:59.805 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:30:59.805 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:30:59.863 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:30:59.864 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:31:00.016 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:31:00.119 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:00.310 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:35 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:31:00.411 INFO:teuthology.orchestra.run.smithi060.stdout:Package libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:31:00.465 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:31:00.467 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2023-11-28T14:31:00.467 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:31:00.493 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:00.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs-devel 2023-11-28T14:31:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:00.789 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:31:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:31:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:00.790 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:31:01.049 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-mirror 2023-11-28T14:31:01.137 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:36 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:31:01.290 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:31:01.291 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:31:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2023-11-28T14:31:01.291 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:31:01.292 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:31:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel x86_64 2:18.0.0-7503.g5537419c.el9 ceph 31 k 2023-11-28T14:31:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:01.292 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:31:01.292 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:31:01.292 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2023-11-28T14:31:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:01.292 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 31 k 2023-11-28T14:31:01.292 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 139 k 2023-11-28T14:31:01.292 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:31:01.544 INFO:teuthology.orchestra.run.smithi060.stdout:libcephfs-devel-18.0.0-7503.g5537419c.el9.x86_6 123 kB/s | 31 kB 00:00 2023-11-28T14:31:01.545 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:31:01.545 INFO:teuthology.orchestra.run.smithi060.stdout:Total 123 kB/s | 31 kB 00:00 2023-11-28T14:31:01.545 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:31:01.548 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:31:01.548 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:31:01.582 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:31:01.582 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:31:01.650 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:26 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:31:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:31:01.778 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:01.804 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:31:01.804 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:31:01.804 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2023-11-28T14:31:01.804 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:31:01.805 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:31:01.805 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror x86_64 2:18.0.0-7503.g5537419c.el9 ceph 3.0 M 2023-11-28T14:31:01.805 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:01.805 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:31:01.805 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:31:01.805 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-28T14:31:01.805 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:01.805 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 3.0 M 2023-11-28T14:31:01.805 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 13 M 2023-11-28T14:31:01.805 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:31:02.156 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:02.267 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-mirror-18.0.0-7503.g5537419c.el9.x86_64.rpm 6.5 MB/s | 3.0 MB 00:00 2023-11-28T14:31:02.268 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:31:02.268 INFO:teuthology.orchestra.run.smithi055.stdout:Total 6.4 MB/s | 3.0 MB 00:00 2023-11-28T14:31:02.268 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:31:02.278 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:31:02.278 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:31:02.354 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:31:02.355 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:31:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:02.442 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:31:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:31:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:02.442 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:31:02.618 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados2 2023-11-28T14:31:02.808 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:31:02.868 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-mirror-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:02.893 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:02.894 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:31:02.894 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-28T14:31:02.894 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T14:31:02.894 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T14:31:02.894 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:03.216 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:38 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:31:03.313 INFO:teuthology.orchestra.run.smithi060.stdout:Package librados2-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:31:03.366 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:31:03.367 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2023-11-28T14:31:03.368 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:31:03.431 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librbd1 2023-11-28T14:31:04.035 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:39 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:31:04.088 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-mirror-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:04.088 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:04.088 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:31:04.088 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:31:04.088 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:04.089 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:31:04.132 INFO:teuthology.orchestra.run.smithi060.stdout:Package librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:31:04.186 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:31:04.187 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2023-11-28T14:31:04.187 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:31:04.256 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rados 2023-11-28T14:31:04.351 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-nbd 2023-11-28T14:31:04.851 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:39 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:31:04.951 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rados-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:31:04.958 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:29 ago on Tue 28 Nov 2023 02:27:35 PM UTC. 2023-11-28T14:31:05.005 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:31:05.006 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2023-11-28T14:31:05.006 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:31:05.072 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rgw 2023-11-28T14:31:05.109 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-28T14:31:05.109 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:31:05.109 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-11-28T14:31:05.110 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:31:05.110 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-28T14:31:05.110 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd x86_64 2:18.0.0-7503.g5537419c.el9 ceph 169 k 2023-11-28T14:31:05.110 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:05.110 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-28T14:31:05.110 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-28T14:31:05.110 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-28T14:31:05.110 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:05.110 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 169 k 2023-11-28T14:31:05.110 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 536 k 2023-11-28T14:31:05.110 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-28T14:31:05.313 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-nbd-18.0.0-7503.g5537419c.el9.x86_64.rpm 836 kB/s | 169 kB 00:00 2023-11-28T14:31:05.313 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:31:05.314 INFO:teuthology.orchestra.run.smithi055.stdout:Total 829 kB/s | 169 kB 00:00 2023-11-28T14:31:05.314 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-28T14:31:05.322 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-28T14:31:05.322 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-28T14:31:05.380 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-28T14:31:05.380 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-28T14:31:05.530 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-28T14:31:05.639 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-nbd-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:05.668 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:40 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:31:05.767 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rgw-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:31:05.821 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:31:05.822 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2023-11-28T14:31:05.822 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:31:05.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-cephfs 2023-11-28T14:31:06.076 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:06.413 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-nbd-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:06.414 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:06.414 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-28T14:31:06.414 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:31:06.414 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:06.414 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-28T14:31:06.515 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:41 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:31:06.612 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-cephfs-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:31:06.650 DEBUG:teuthology.parallel:result is None 2023-11-28T14:31:06.666 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:31:06.667 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2023-11-28T14:31:06.667 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:31:06.730 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rbd 2023-11-28T14:31:07.329 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:42 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:31:07.430 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rbd-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:31:07.484 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:31:07.485 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2023-11-28T14:31:07.485 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:31:07.564 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-fuse 2023-11-28T14:31:08.164 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:43 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:31:08.315 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:31:08.316 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:31:08.316 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2023-11-28T14:31:08.316 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:31:08.316 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:31:08.316 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse x86_64 2:18.0.0-7503.g5537419c.el9 ceph 86 k 2023-11-28T14:31:08.316 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:08.316 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:31:08.316 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:31:08.316 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2023-11-28T14:31:08.316 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:08.316 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 86 k 2023-11-28T14:31:08.317 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 256 k 2023-11-28T14:31:08.317 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:31:08.664 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-fuse-18.0.0-7503.g5537419c.el9.x86_64.rpm 247 kB/s | 86 kB 00:00 2023-11-28T14:31:08.664 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:31:08.664 INFO:teuthology.orchestra.run.smithi060.stdout:Total 246 kB/s | 86 kB 00:00 2023-11-28T14:31:08.664 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:31:08.672 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:31:08.672 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:31:08.729 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:31:08.730 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:31:08.869 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:31:08.972 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:13.274 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:13.274 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:13.274 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:13.274 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:31:13.274 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:31:13.274 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:13.274 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:31:13.275 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-mirror 2023-11-28T14:31:13.888 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:48 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:31:14.042 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:31:14.098 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:31:14.098 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2023-11-28T14:31:14.098 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:31:14.098 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:31:14.098 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror x86_64 2:18.0.0-7503.g5537419c.el9 ceph 3.0 M 2023-11-28T14:31:14.099 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:14.099 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:31:14.099 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:31:14.099 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2023-11-28T14:31:14.099 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:14.099 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 3.0 M 2023-11-28T14:31:14.099 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 13 M 2023-11-28T14:31:14.099 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:31:14.592 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-mirror-18.0.0-7503.g5537419c.el9.x86_64.rpm 5.4 MB/s | 3.0 MB 00:00 2023-11-28T14:31:14.592 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:31:14.593 INFO:teuthology.orchestra.run.smithi060.stdout:Total 5.4 MB/s | 3.0 MB 00:00 2023-11-28T14:31:14.593 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:31:14.603 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:31:14.603 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:31:14.679 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:31:14.680 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:31:15.130 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:31:15.201 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:15.226 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:15.226 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:31:15.226 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-28T14:31:15.227 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T14:31:15.227 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T14:31:15.227 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:16.366 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:16.366 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:16.366 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:31:16.366 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:31:16.366 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:16.366 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:31:16.597 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-nbd 2023-11-28T14:31:17.206 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:52 ago on Tue 28 Nov 2023 02:27:25 PM UTC. 2023-11-28T14:31:17.358 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-11-28T14:31:17.359 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:31:17.359 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2023-11-28T14:31:17.359 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:31:17.359 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-11-28T14:31:17.359 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd x86_64 2:18.0.0-7503.g5537419c.el9 ceph 169 k 2023-11-28T14:31:17.359 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:17.359 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-11-28T14:31:17.360 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-11-28T14:31:17.360 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2023-11-28T14:31:17.360 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:17.360 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 169 k 2023-11-28T14:31:17.360 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 536 k 2023-11-28T14:31:17.360 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-11-28T14:31:17.970 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-nbd-18.0.0-7503.g5537419c.el9.x86_64.rpm 277 kB/s | 169 kB 00:00 2023-11-28T14:31:17.971 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:31:17.971 INFO:teuthology.orchestra.run.smithi060.stdout:Total 276 kB/s | 169 kB 00:00 2023-11-28T14:31:17.971 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-11-28T14:31:17.980 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-11-28T14:31:17.980 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-11-28T14:31:18.041 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-11-28T14:31:18.041 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-11-28T14:31:18.203 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-11-28T14:31:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:18.729 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:19.023 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:31:19.023 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:19.023 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-11-28T14:31:19.024 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:31:19.024 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:19.024 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-11-28T14:31:19.276 DEBUG:teuthology.parallel:result is None 2023-11-28T14:31:19.277 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=5537419ca37a48f0ea28d7872ae1f14ccf1a953b 2023-11-28T14:31:19.495 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-28T14:31:19.521 INFO:teuthology.orchestra.run.smithi055.stdout:18.0.0-7503.g5537419c.el9 2023-11-28T14:31:19.521 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7503.g5537419c.el9 2023-11-28T14:31:19.522 INFO:teuthology.task.install:The correct ceph version 18.0.0-7503.g5537419c is installed. 2023-11-28T14:31:19.522 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=5537419ca37a48f0ea28d7872ae1f14ccf1a953b 2023-11-28T14:31:19.776 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-28T14:31:19.801 INFO:teuthology.orchestra.run.smithi060.stdout:18.0.0-7503.g5537419c.el9 2023-11-28T14:31:19.801 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7503.g5537419c.el9 2023-11-28T14:31:19.802 INFO:teuthology.task.install:The correct ceph version 18.0.0-7503.g5537419c is installed. 2023-11-28T14:31:19.802 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-28T14:31:19.802 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:19.802 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T14:31:19.841 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:31:19.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T14:31:19.877 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-28T14:31:19.878 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:19.878 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T14:31:19.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T14:31:19.988 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:31:19.988 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T14:31:20.022 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T14:31:20.094 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-28T14:31:20.095 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:20.095 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T14:31:20.127 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T14:31:20.200 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:31:20.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T14:31:20.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T14:31:20.305 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-28T14:31:20.305 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:20.305 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T14:31:20.337 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T14:31:20.410 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:31:20.410 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T14:31:20.441 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T14:31:20.514 INFO:teuthology.run_tasks:Running task ceph... 2023-11-28T14:31:20.581 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-28T14:31:20.581 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 777 /var/log/ceph 2023-11-28T14:31:20.584 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2023-11-28T14:31:20.614 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-28T14:31:20.615 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-28T14:31:20.656 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-28T14:31:20.686 INFO:tasks.ceph:Creating extra log directories... 2023-11-28T14:31:20.687 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-28T14:31:20.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-28T14:31:20.769 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-28T14:31:20.770 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-28T14:31:20.770 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470178', 'branch': 'wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced}', 'email': None, 'first_in_suite': False, 'job_id': '7470178', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'b9b6c42eed672399c3d01aa72f6cda1a665a3c35'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9581, 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'b9b6c42eed672399c3d01aa72f6cda1a665a3c35', 'targets': {'smithi055.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ4lVYdl6BZ/JhGhiPVEk5zjACkVuQ0SXB2blbo1W44KtcLD3nJ9YmK0QJZI9aMOQpZCZMGXhyZqff2YmxQGihM=', 'smithi060.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMujrKC3zNd9bMVWC1Jc8fC/Rj6GOAlVisVeGkXaMfmsuFAl3j+cc/QV7CJ9l+/Jt1/vjKg46VKPe4FLB38t5m4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b', 'wait-for-scrub': False, 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'rados': {'balance_reads': True, 'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '5ae09a5cc7064ca2bf994df19f3006b79794789a', 'timestamp': '2023-11-28_14:04:23', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-28T14:31:20.770 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-28T14:31:20.800 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-28T14:31:20.815 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m0777 -- /var/run/ceph 2023-11-28T14:31:20.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2023-11-28T14:31:20.887 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:20.887 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T14:31:20.941 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T14:31:20.941 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_1 2023-11-28T14:31:20.996 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T14:31:20.997 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T14:31:20.997 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-11-28T14:31:20.997 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T14:31:20.997 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T14:31:20.997 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2023-11-28 14:31:03.127759435 +0000 2023-11-28T14:31:20.997 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2023-11-28 14:26:27.247929605 +0000 2023-11-28T14:31:20.997 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2023-11-28 14:26:27.247929605 +0000 2023-11-28T14:31:20.997 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2023-11-28 14:26:27.247929605 +0000 2023-11-28T14:31:20.998 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T14:31:21.066 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2023-11-28T14:31:21.067 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2023-11-28T14:31:21.067 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000298093 s, 1.7 MB/s 2023-11-28T14:31:21.068 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T14:31:21.124 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_2 2023-11-28T14:31:21.180 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T14:31:21.180 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T14:31:21.181 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-11-28T14:31:21.181 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T14:31:21.181 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T14:31:21.181 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2023-11-28 14:31:03.128759415 +0000 2023-11-28T14:31:21.181 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2023-11-28 14:26:27.775918933 +0000 2023-11-28T14:31:21.181 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2023-11-28 14:26:27.775918933 +0000 2023-11-28T14:31:21.181 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2023-11-28 14:26:27.775918933 +0000 2023-11-28T14:31:21.181 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T14:31:21.249 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2023-11-28T14:31:21.249 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2023-11-28T14:31:21.249 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000273692 s, 1.9 MB/s 2023-11-28T14:31:21.251 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T14:31:21.307 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_3 2023-11-28T14:31:21.362 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T14:31:21.362 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T14:31:21.362 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-11-28T14:31:21.362 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T14:31:21.363 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T14:31:21.363 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2023-11-28 14:31:03.128759415 +0000 2023-11-28T14:31:21.363 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2023-11-28 14:26:28.280908725 +0000 2023-11-28T14:31:21.363 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2023-11-28 14:26:28.280908725 +0000 2023-11-28T14:31:21.363 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2023-11-28 14:26:28.280908725 +0000 2023-11-28T14:31:21.363 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T14:31:21.431 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2023-11-28T14:31:21.431 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2023-11-28T14:31:21.431 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000263776 s, 1.9 MB/s 2023-11-28T14:31:21.433 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T14:31:21.489 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_4 2023-11-28T14:31:21.544 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T14:31:21.544 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T14:31:21.545 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-11-28T14:31:21.545 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T14:31:21.545 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T14:31:21.545 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2023-11-28 14:31:03.128759415 +0000 2023-11-28T14:31:21.545 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2023-11-28 14:26:28.801898195 +0000 2023-11-28T14:31:21.545 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2023-11-28 14:26:28.801898195 +0000 2023-11-28T14:31:21.545 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2023-11-28 14:26:28.801898195 +0000 2023-11-28T14:31:21.545 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T14:31:21.613 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2023-11-28T14:31:21.614 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2023-11-28T14:31:21.614 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000315591 s, 1.6 MB/s 2023-11-28T14:31:21.615 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T14:31:21.671 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-11-28T14:31:21.672 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:31:21.672 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T14:31:21.690 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T14:31:21.691 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2023-11-28T14:31:21.749 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T14:31:21.749 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T14:31:21.749 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-11-28T14:31:21.749 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T14:31:21.749 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T14:31:21.750 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2023-11-28 14:31:15.456261235 +0000 2023-11-28T14:31:21.750 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2023-11-28 14:26:27.234950077 +0000 2023-11-28T14:31:21.750 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2023-11-28 14:26:27.234950077 +0000 2023-11-28T14:31:21.750 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2023-11-28 14:26:27.234950077 +0000 2023-11-28T14:31:21.750 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T14:31:21.822 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2023-11-28T14:31:21.823 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2023-11-28T14:31:21.823 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000114374 s, 4.5 MB/s 2023-11-28T14:31:21.824 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T14:31:21.882 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2023-11-28T14:31:21.939 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T14:31:21.939 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T14:31:21.939 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-11-28T14:31:21.939 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T14:31:21.939 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T14:31:21.940 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2023-11-28 14:31:15.457261225 +0000 2023-11-28T14:31:21.940 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2023-11-28 14:26:27.737945133 +0000 2023-11-28T14:31:21.940 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2023-11-28 14:26:27.737945133 +0000 2023-11-28T14:31:21.940 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2023-11-28 14:26:27.737945133 +0000 2023-11-28T14:31:21.940 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T14:31:22.010 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2023-11-28T14:31:22.011 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2023-11-28T14:31:22.011 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000109812 s, 4.7 MB/s 2023-11-28T14:31:22.012 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T14:31:22.069 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2023-11-28T14:31:22.125 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T14:31:22.126 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T14:31:22.126 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-11-28T14:31:22.126 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T14:31:22.126 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T14:31:22.126 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2023-11-28 14:31:15.457261225 +0000 2023-11-28T14:31:22.126 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2023-11-28 14:26:28.268939914 +0000 2023-11-28T14:31:22.126 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2023-11-28 14:26:28.268939914 +0000 2023-11-28T14:31:22.126 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2023-11-28 14:26:28.268939914 +0000 2023-11-28T14:31:22.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T14:31:22.197 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2023-11-28T14:31:22.197 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2023-11-28T14:31:22.198 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000103445 s, 4.9 MB/s 2023-11-28T14:31:22.199 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T14:31:22.256 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2023-11-28T14:31:22.313 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T14:31:22.313 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T14:31:22.313 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-11-28T14:31:22.313 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T14:31:22.313 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T14:31:22.313 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2023-11-28 14:31:15.457261225 +0000 2023-11-28T14:31:22.313 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2023-11-28 14:26:28.778934902 +0000 2023-11-28T14:31:22.314 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2023-11-28 14:26:28.778934902 +0000 2023-11-28T14:31:22.314 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2023-11-28 14:26:28.778934902 +0000 2023-11-28T14:31:22.314 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T14:31:22.383 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2023-11-28T14:31:22.383 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2023-11-28T14:31:22.383 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 8.7321e-05 s, 5.9 MB/s 2023-11-28T14:31:22.384 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T14:31:22.440 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-11-28T14:31:22.440 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi055.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2023-11-28T14:31:22.440 INFO:tasks.ceph:Generating config... 2023-11-28T14:31:22.441 INFO:tasks.ceph:[osd] debug monc = 20 2023-11-28T14:31:22.442 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-28T14:31:22.442 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-11-28T14:31:22.442 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-11-28T14:31:22.442 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-28T14:31:22.442 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-11-28T14:31:22.442 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-11-28T14:31:22.442 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-28T14:31:22.442 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-28T14:31:22.442 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-28T14:31:22.442 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-28T14:31:22.442 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-28T14:31:22.443 INFO:tasks.ceph:[osd] err to stderr = True 2023-11-28T14:31:22.443 INFO:tasks.ceph:[osd] log flush on exit = True 2023-11-28T14:31:22.443 INFO:tasks.ceph:[osd] log to file = False 2023-11-28T14:31:22.443 INFO:tasks.ceph:[osd] log to stderr = True 2023-11-28T14:31:22.443 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-28T14:31:22.443 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-28T14:31:22.443 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-28T14:31:22.443 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-11-28T14:31:22.443 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2023-11-28T14:31:22.443 INFO:tasks.ceph:[osd] osd delete sleep = 1 2023-11-28T14:31:22.443 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-28T14:31:22.444 INFO:tasks.ceph:[osd] osd max backfills = 3 2023-11-28T14:31:22.444 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-11-28T14:31:22.444 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-28T14:31:22.444 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2023-11-28T14:31:22.444 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2023-11-28T14:31:22.444 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-11-28T14:31:22.444 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2023-11-28T14:31:22.444 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-11-28T14:31:22.444 INFO:tasks.ceph:[global] ms client mode = crc 2023-11-28T14:31:22.444 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-11-28T14:31:22.444 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-11-28T14:31:22.445 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-11-28T14:31:22.445 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-11-28T14:31:22.445 INFO:tasks.ceph:[global] ms service mode = crc 2023-11-28T14:31:22.445 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2023-11-28T14:31:22.445 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2023-11-28T14:31:22.445 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-28T14:31:22.445 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-28T14:31:22.445 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-28T14:31:22.445 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-28T14:31:22.446 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-28T14:31:22.446 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-11-28T14:31:22.446 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-11-28T14:31:22.446 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-11-28T14:31:22.446 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-11-28T14:31:22.446 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-11-28T14:31:22.446 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-11-28T14:31:22.447 INFO:tasks.ceph:Setting up mon.a... 2023-11-28T14:31:22.447 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-28T14:31:22.520 INFO:teuthology.orchestra.run.smithi055.stdout:creating /etc/ceph/ceph.keyring 2023-11-28T14:31:22.523 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-28T14:31:22.581 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T14:31:22.612 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.55')] 2023-11-28T14:31:22.612 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.55', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-11-28T14:31:22.613 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:22.613 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T14:31:22.667 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.55 --print /home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:31:22.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:31:22.751+0000 7f083ce2ae80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:22.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:31:22.751+0000 7f083ce2ae80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:22.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:31:22.752+0000 7f083ce2ae80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:22.754 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:31:22.754 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: generated fsid 8639ce7e-f555-422d-b8f1-79137ef3334b 2023-11-28T14:31:22.755 INFO:teuthology.orchestra.run.smithi055.stdout:setting min_mon_release = pacific 2023-11-28T14:31:22.755 INFO:teuthology.orchestra.run.smithi055.stdout:epoch 0 2023-11-28T14:31:22.755 INFO:teuthology.orchestra.run.smithi055.stdout:fsid 8639ce7e-f555-422d-b8f1-79137ef3334b 2023-11-28T14:31:22.755 INFO:teuthology.orchestra.run.smithi055.stdout:last_changed 2023-11-28T14:31:22.752976+0000 2023-11-28T14:31:22.755 INFO:teuthology.orchestra.run.smithi055.stdout:created 2023-11-28T14:31:22.752976+0000 2023-11-28T14:31:22.755 INFO:teuthology.orchestra.run.smithi055.stdout:min_mon_release 16 (pacific) 2023-11-28T14:31:22.756 INFO:teuthology.orchestra.run.smithi055.stdout:election_strategy: 1 2023-11-28T14:31:22.756 INFO:teuthology.orchestra.run.smithi055.stdout:0: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.a 2023-11-28T14:31:22.756 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-11-28T14:31:22.756 DEBUG:teuthology.orchestra.run.smithi055:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T14:31:22.771 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8639ce7e-f555-422d-b8f1-79137ef3334b... 2023-11-28T14:31:22.772 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-28T14:31:22.813 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-28T14:31:22.871 INFO:teuthology.orchestra.run.smithi055.stdout:[global] 2023-11-28T14:31:22.871 INFO:teuthology.orchestra.run.smithi055.stdout: chdir = "" 2023-11-28T14:31:22.871 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-28T14:31:22.871 INFO:teuthology.orchestra.run.smithi055.stdout: auth supported = cephx 2023-11-28T14:31:22.871 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.871 INFO:teuthology.orchestra.run.smithi055.stdout: filestore xattr use omap = true 2023-11-28T14:31:22.872 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.872 INFO:teuthology.orchestra.run.smithi055.stdout: mon clock drift allowed = 1.000 2023-11-28T14:31:22.872 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.872 INFO:teuthology.orchestra.run.smithi055.stdout: osd crush chooseleaf type = 0 2023-11-28T14:31:22.872 INFO:teuthology.orchestra.run.smithi055.stdout: auth debug = true 2023-11-28T14:31:22.872 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.872 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on old message = true 2023-11-28T14:31:22.872 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on bug = true 2023-11-28T14:31:22.872 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.872 INFO:teuthology.orchestra.run.smithi055.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-28T14:31:22.872 INFO:teuthology.orchestra.run.smithi055.stdout: mon pg warn max object skew = 0 2023-11-28T14:31:22.872 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.873 INFO:teuthology.orchestra.run.smithi055.stdout: # disable pg_autoscaler by default for new pools 2023-11-28T14:31:22.873 INFO:teuthology.orchestra.run.smithi055.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-28T14:31:22.873 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.873 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default size = 2 2023-11-28T14:31:22.873 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.873 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow primary affinity = true 2023-11-28T14:31:22.873 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow pg remap = true 2023-11-28T14:31:22.873 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on legacy crush tunables = false 2023-11-28T14:31:22.873 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on crush straw calc version zero = false 2023-11-28T14:31:22.873 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on no sortbitwise = false 2023-11-28T14:31:22.873 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on osd down out interval zero = false 2023-11-28T14:31:22.873 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on too few osds = false 2023-11-28T14:31:22.874 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-28T14:31:22.874 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-28T14:31:22.874 INFO:teuthology.orchestra.run.smithi055.stdout: mon_allow_pool_size_one = true 2023-11-28T14:31:22.874 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.874 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-28T14:31:22.874 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.874 INFO:teuthology.orchestra.run.smithi055.stdout: osd default data pool replay window = 5 2023-11-28T14:31:22.874 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.874 INFO:teuthology.orchestra.run.smithi055.stdout: mon allow pool delete = true 2023-11-28T14:31:22.874 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.875 INFO:teuthology.orchestra.run.smithi055.stdout: mon cluster log file level = debug 2023-11-28T14:31:22.875 INFO:teuthology.orchestra.run.smithi055.stdout: debug asserts on shutdown = true 2023-11-28T14:31:22.875 INFO:teuthology.orchestra.run.smithi055.stdout: mon health detail to clog = false 2023-11-28T14:31:22.875 INFO:teuthology.orchestra.run.smithi055.stdout: mon host = 172.21.15.55 2023-11-28T14:31:22.875 INFO:teuthology.orchestra.run.smithi055.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-11-28T14:31:22.875 INFO:teuthology.orchestra.run.smithi055.stdout: ms client mode = crc 2023-11-28T14:31:22.875 INFO:teuthology.orchestra.run.smithi055.stdout: ms cluster mode = crc 2023-11-28T14:31:22.875 INFO:teuthology.orchestra.run.smithi055.stdout: ms mon client mode = crc 2023-11-28T14:31:22.875 INFO:teuthology.orchestra.run.smithi055.stdout: ms mon cluster mode = crc 2023-11-28T14:31:22.876 INFO:teuthology.orchestra.run.smithi055.stdout: ms mon service mode = crc 2023-11-28T14:31:22.876 INFO:teuthology.orchestra.run.smithi055.stdout: ms service mode = crc 2023-11-28T14:31:22.876 INFO:teuthology.orchestra.run.smithi055.stdout: osd_pool_default_min_size = 2 2023-11-28T14:31:22.876 INFO:teuthology.orchestra.run.smithi055.stdout: osd_pool_default_size = 3 2023-11-28T14:31:22.876 INFO:teuthology.orchestra.run.smithi055.stdout: fsid = 8639ce7e-f555-422d-b8f1-79137ef3334b 2023-11-28T14:31:22.876 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.876 INFO:teuthology.orchestra.run.smithi055.stdout:[osd] 2023-11-28T14:31:22.876 INFO:teuthology.orchestra.run.smithi055.stdout: osd journal size = 100 2023-11-28T14:31:22.876 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.876 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub load threshold = 5.0 2023-11-28T14:31:22.876 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub max interval = 120 2023-11-28T14:31:22.877 INFO:teuthology.orchestra.run.smithi055.stdout: osd mclock profile = high_recovery_ops 2023-11-28T14:31:22.877 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.877 INFO:teuthology.orchestra.run.smithi055.stdout: osd recover clone overlap = true 2023-11-28T14:31:22.877 INFO:teuthology.orchestra.run.smithi055.stdout: osd recovery max chunk = 1048576 2023-11-28T14:31:22.877 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.877 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug shutdown = true 2023-11-28T14:31:22.877 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug op order = true 2023-11-28T14:31:22.877 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug verify stray on activate = true 2023-11-28T14:31:22.877 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug trim objects = true 2023-11-28T14:31:22.877 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.877 INFO:teuthology.orchestra.run.smithi055.stdout: osd open classes on start = true 2023-11-28T14:31:22.877 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug pg log writeout = true 2023-11-28T14:31:22.878 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.878 INFO:teuthology.orchestra.run.smithi055.stdout: osd deep scrub update digest min age = 30 2023-11-28T14:31:22.878 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.878 INFO:teuthology.orchestra.run.smithi055.stdout: osd map max advance = 10 2023-11-28T14:31:22.878 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.878 INFO:teuthology.orchestra.run.smithi055.stdout: journal zero on create = true 2023-11-28T14:31:22.878 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.878 INFO:teuthology.orchestra.run.smithi055.stdout: filestore ondisk finisher threads = 3 2023-11-28T14:31:22.878 INFO:teuthology.orchestra.run.smithi055.stdout: filestore apply finisher threads = 3 2023-11-28T14:31:22.878 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.878 INFO:teuthology.orchestra.run.smithi055.stdout: bdev debug aio = true 2023-11-28T14:31:22.878 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug misdirected ops = true 2023-11-28T14:31:22.879 INFO:teuthology.orchestra.run.smithi055.stdout: debug monc = 20 2023-11-28T14:31:22.879 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore block size = 96636764160 2023-11-28T14:31:22.879 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore compression algorithm = snappy 2023-11-28T14:31:22.879 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore compression mode = aggressive 2023-11-28T14:31:22.879 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore fsck on mount = True 2023-11-28T14:31:22.879 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore rocksdb cf = False 2023-11-28T14:31:22.879 INFO:teuthology.orchestra.run.smithi055.stdout: crimson seastar smp = 3 2023-11-28T14:31:22.879 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluefs = 20 2023-11-28T14:31:22.879 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluestore = 20 2023-11-28T14:31:22.879 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2023-11-28T14:31:22.880 INFO:teuthology.orchestra.run.smithi055.stdout: debug osd = 20 2023-11-28T14:31:22.880 INFO:teuthology.orchestra.run.smithi055.stdout: debug rocksdb = 10 2023-11-28T14:31:22.880 INFO:teuthology.orchestra.run.smithi055.stdout: err to stderr = True 2023-11-28T14:31:22.880 INFO:teuthology.orchestra.run.smithi055.stdout: log flush on exit = True 2023-11-28T14:31:22.880 INFO:teuthology.orchestra.run.smithi055.stdout: log to file = False 2023-11-28T14:31:22.880 INFO:teuthology.orchestra.run.smithi055.stdout: log to stderr = True 2023-11-28T14:31:22.880 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd backfillfull_ratio = 0.85 2023-11-28T14:31:22.880 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd full ratio = 0.9 2023-11-28T14:31:22.880 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd nearfull ratio = 0.8 2023-11-28T14:31:22.880 INFO:teuthology.orchestra.run.smithi055.stdout: osd blocked scrub grace period = 3600 2023-11-28T14:31:22.881 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug reject backfill probability = 0.3 2023-11-28T14:31:22.881 INFO:teuthology.orchestra.run.smithi055.stdout: osd delete sleep = 1 2023-11-28T14:31:22.881 INFO:teuthology.orchestra.run.smithi055.stdout: osd failsafe full ratio = 0.95 2023-11-28T14:31:22.881 INFO:teuthology.orchestra.run.smithi055.stdout: osd max backfills = 3 2023-11-28T14:31:22.881 INFO:teuthology.orchestra.run.smithi055.stdout: osd max markdown count = 1000 2023-11-28T14:31:22.881 INFO:teuthology.orchestra.run.smithi055.stdout: osd objectstore = bluestore 2023-11-28T14:31:22.881 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub min interval = 60 2023-11-28T14:31:22.881 INFO:teuthology.orchestra.run.smithi055.stdout: osd shutdown pgref assert = True 2023-11-28T14:31:22.881 INFO:teuthology.orchestra.run.smithi055.stdout: osd snap trim sleep = 2 2023-11-28T14:31:22.881 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.881 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr] 2023-11-28T14:31:22.881 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2023-11-28T14:31:22.882 INFO:teuthology.orchestra.run.smithi055.stdout: debug mgr = 20 2023-11-28T14:31:22.882 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2023-11-28T14:31:22.882 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2023-11-28T14:31:22.882 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2023-11-28T14:31:22.882 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2023-11-28T14:31:22.882 INFO:teuthology.orchestra.run.smithi055.stdout: mgr/telemetry/nag = false 2023-11-28T14:31:22.882 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.882 INFO:teuthology.orchestra.run.smithi055.stdout:[mon] 2023-11-28T14:31:22.882 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2023-11-28T14:31:22.882 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2023-11-28T14:31:22.883 INFO:teuthology.orchestra.run.smithi055.stdout: debug paxos = 20 2023-11-28T14:31:22.883 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2023-11-28T14:31:22.883 INFO:teuthology.orchestra.run.smithi055.stdout: mon data avail warn = 5 2023-11-28T14:31:22.883 INFO:teuthology.orchestra.run.smithi055.stdout: mon mgr mkfs grace = 240 2023-11-28T14:31:22.883 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2023-11-28T14:31:22.883 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd reporter subtree level = osd 2023-11-28T14:31:22.883 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd prime pg temp = true 2023-11-28T14:31:22.883 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2023-11-28T14:31:22.883 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.883 INFO:teuthology.orchestra.run.smithi055.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-28T14:31:22.884 INFO:teuthology.orchestra.run.smithi055.stdout: auth mon ticket ttl = 660 # 11m 2023-11-28T14:31:22.884 INFO:teuthology.orchestra.run.smithi055.stdout: auth service ticket ttl = 240 # 4m 2023-11-28T14:31:22.884 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.884 INFO:teuthology.orchestra.run.smithi055.stdout: # don't complain about insecure global_id in the test suite 2023-11-28T14:31:22.884 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-28T14:31:22.884 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-28T14:31:22.884 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.884 INFO:teuthology.orchestra.run.smithi055.stdout: # 1m isn't quite enough 2023-11-28T14:31:22.884 INFO:teuthology.orchestra.run.smithi055.stdout: mon_down_mkfs_grace = 2m 2023-11-28T14:31:22.884 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.884 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_filestore_osds = false 2023-11-28T14:31:22.885 INFO:teuthology.orchestra.run.smithi055.stdout: mon min osdmap epochs = 50 2023-11-28T14:31:22.885 INFO:teuthology.orchestra.run.smithi055.stdout: mon osdmap full prune interval = 2 2023-11-28T14:31:22.885 INFO:teuthology.orchestra.run.smithi055.stdout: mon osdmap full prune min = 15 2023-11-28T14:31:22.885 INFO:teuthology.orchestra.run.smithi055.stdout: mon osdmap full prune txsize = 2 2023-11-28T14:31:22.885 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default crimson = True 2023-11-28T14:31:22.885 INFO:teuthology.orchestra.run.smithi055.stdout: paxos service trim min = 10 2023-11-28T14:31:22.885 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:22.885 INFO:teuthology.orchestra.run.smithi055.stdout:[client] 2023-11-28T14:31:22.885 INFO:teuthology.orchestra.run.smithi055.stdout: rgw cache enabled = true 2023-11-28T14:31:22.885 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable ops log = true 2023-11-28T14:31:22.886 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable usage log = true 2023-11-28T14:31:22.886 INFO:teuthology.orchestra.run.smithi055.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-28T14:31:22.886 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-28T14:31:22.886 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.a] 2023-11-28T14:31:22.886 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2023-11-28T14:31:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2023-11-28T14:31:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-28T14:31:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2023-11-28T14:31:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2023-11-28T14:31:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2023-11-28T14:31:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2023-11-28T14:31:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2023-11-28T14:31:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2023-11-28T14:31:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2023-11-28T14:31:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-28T14:31:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2023-11-28T14:31:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2023-11-28T14:31:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-28T14:31:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2023-11-28T14:31:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2023-11-28T14:31:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2023-11-28T14:31:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2023-11-28T14:31:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2023-11-28T14:31:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2023-11-28T14:31:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2023-11-28T14:31:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2023-11-28T14:31:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-28T14:31:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-28T14:31:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2023-11-28T14:31:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-28T14:31:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2023-11-28T14:31:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2023-11-28T14:31:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2023-11-28T14:31:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2023-11-28T14:31:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2023-11-28T14:31:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = 172.21.15.55 2023-11-28T14:31:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-11-28T14:31:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: ms client mode = crc 2023-11-28T14:31:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: ms cluster mode = crc 2023-11-28T14:31:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: ms mon client mode = crc 2023-11-28T14:31:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: ms mon cluster mode = crc 2023-11-28T14:31:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: ms mon service mode = crc 2023-11-28T14:31:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: ms service mode = crc 2023-11-28T14:31:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_min_size = 2 2023-11-28T14:31:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_size = 3 2023-11-28T14:31:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = 8639ce7e-f555-422d-b8f1-79137ef3334b 2023-11-28T14:31:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.891 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2023-11-28T14:31:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2023-11-28T14:31:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2023-11-28T14:31:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 120 2023-11-28T14:31:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock profile = high_recovery_ops 2023-11-28T14:31:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2023-11-28T14:31:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2023-11-28T14:31:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2023-11-28T14:31:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2023-11-28T14:31:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2023-11-28T14:31:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug trim objects = true 2023-11-28T14:31:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2023-11-28T14:31:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2023-11-28T14:31:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2023-11-28T14:31:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2023-11-28T14:31:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2023-11-28T14:31:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2023-11-28T14:31:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2023-11-28T14:31:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2023-11-28T14:31:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2023-11-28T14:31:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: debug monc = 20 2023-11-28T14:31:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore block size = 96636764160 2023-11-28T14:31:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore compression algorithm = snappy 2023-11-28T14:31:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore compression mode = aggressive 2023-11-28T14:31:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore fsck on mount = True 2023-11-28T14:31:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore rocksdb cf = False 2023-11-28T14:31:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: crimson seastar smp = 3 2023-11-28T14:31:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluefs = 20 2023-11-28T14:31:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluestore = 20 2023-11-28T14:31:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2023-11-28T14:31:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2023-11-28T14:31:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: debug rocksdb = 10 2023-11-28T14:31:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: err to stderr = True 2023-11-28T14:31:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: log flush on exit = True 2023-11-28T14:31:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: log to file = False 2023-11-28T14:31:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: log to stderr = True 2023-11-28T14:31:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd backfillfull_ratio = 0.85 2023-11-28T14:31:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd full ratio = 0.9 2023-11-28T14:31:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd nearfull ratio = 0.8 2023-11-28T14:31:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: osd blocked scrub grace period = 3600 2023-11-28T14:31:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug reject backfill probability = 0.3 2023-11-28T14:31:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: osd delete sleep = 1 2023-11-28T14:31:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: osd failsafe full ratio = 0.95 2023-11-28T14:31:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: osd max backfills = 3 2023-11-28T14:31:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: osd max markdown count = 1000 2023-11-28T14:31:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = bluestore 2023-11-28T14:31:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub min interval = 60 2023-11-28T14:31:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: osd shutdown pgref assert = True 2023-11-28T14:31:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: osd snap trim sleep = 2 2023-11-28T14:31:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.897 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2023-11-28T14:31:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2023-11-28T14:31:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2023-11-28T14:31:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2023-11-28T14:31:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2023-11-28T14:31:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2023-11-28T14:31:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2023-11-28T14:31:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2023-11-28T14:31:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.897 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2023-11-28T14:31:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2023-11-28T14:31:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2023-11-28T14:31:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2023-11-28T14:31:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2023-11-28T14:31:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2023-11-28T14:31:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2023-11-28T14:31:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2023-11-28T14:31:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2023-11-28T14:31:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2023-11-28T14:31:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2023-11-28T14:31:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-28T14:31:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2023-11-28T14:31:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2023-11-28T14:31:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2023-11-28T14:31:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-28T14:31:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-28T14:31:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: # 1m isn't quite enough 2023-11-28T14:31:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: mon_down_mkfs_grace = 2m 2023-11-28T14:31:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_filestore_osds = false 2023-11-28T14:31:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: mon min osdmap epochs = 50 2023-11-28T14:31:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: mon osdmap full prune interval = 2 2023-11-28T14:31:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: mon osdmap full prune min = 15 2023-11-28T14:31:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: mon osdmap full prune txsize = 2 2023-11-28T14:31:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default crimson = True 2023-11-28T14:31:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: paxos service trim min = 10 2023-11-28T14:31:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-11-28T14:31:22.900 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2023-11-28T14:31:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2023-11-28T14:31:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2023-11-28T14:31:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2023-11-28T14:31:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-28T14:31:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-28T14:31:22.901 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2023-11-28T14:31:22.902 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-28T14:31:22.902 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-28T14:31:23.007 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-28T14:31:23.008 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:23.008 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-28T14:31:23.023 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:23.023 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-28T14:31:23.078 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi055.front.sepia.ceph.com 2023-11-28T14:31:23.078 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:23.078 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-28T14:31:23.078 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T14:31:23.165 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:23.165 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:31:23.221 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2023-11-28T14:31:23.222 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:31:23.222 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-28T14:31:23.222 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T14:31:23.272 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:31:23.272 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:31:23.328 INFO:tasks.ceph:Setting up mon nodes... 2023-11-28T14:31:23.328 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-28T14:31:23.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T14:31:23.449 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T14:31:23.452 INFO:tasks.ceph:Setting up mds nodes... 2023-11-28T14:31:23.452 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-28T14:31:23.453 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-28T14:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-28T14:31:23.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-11-28T14:31:23.577 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-11-28T14:31:23.595 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-28T14:31:23.595 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi055.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2023-11-28T14:31:23.596 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-28T14:31:23.626 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-11-28T14:31:23.627 INFO:tasks.ceph:role: osd.0 2023-11-28T14:31:23.627 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi055.front.sepia.ceph.com 2023-11-28T14:31:23.627 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-28T14:31:23.725 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T14:31:23.725 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T14:31:23.725 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T14:31:23.725 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T14:31:23.725 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T14:31:23.725 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2023-11-28T14:31:23.725 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T14:31:23.726 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T14:31:23.726 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T14:31:23.726 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T14:31:25.064 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2023-11-28T14:31:25.066 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2023-11-28T14:31:25.067 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-28T14:31:25.110 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-28T14:31:25.190 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-28T14:31:25.262 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-11-28T14:31:25.263 INFO:tasks.ceph:role: osd.1 2023-11-28T14:31:25.263 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi055.front.sepia.ceph.com 2023-11-28T14:31:25.263 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-28T14:31:25.363 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T14:31:25.363 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T14:31:25.363 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T14:31:25.364 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T14:31:25.364 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T14:31:25.364 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2023-11-28T14:31:25.364 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T14:31:25.364 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T14:31:25.364 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T14:31:25.365 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T14:31:26.678 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2023-11-28T14:31:26.680 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2023-11-28T14:31:26.680 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-28T14:31:26.719 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-28T14:31:26.796 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:31:27.357 INFO:teuthology.orchestra.run.smithi055.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:31:27.358 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:27,357 seastar - Reactor backend: linux-aio 2023-11-28T14:31:27.377 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:27,377 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:31:27.378 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:27,377 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:31:27.378 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:27,377 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:31:27.378 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:27,377 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:31:27.380 INFO:teuthology.orchestra.run.smithi055.stderr:==80638==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:31:27.386 INFO:teuthology.orchestra.run.smithi055.stderr:==80638==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffcb51b000; bottom 0x7f0f9aa27000; size: 0x00f030af4000 (1031608942592) 2023-11-28T14:31:27.386 INFO:teuthology.orchestra.run.smithi055.stderr:False positive error reports may follow 2023-11-28T14:31:27.386 INFO:teuthology.orchestra.run.smithi055.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:31:27.412 INFO:teuthology.orchestra.run.smithi055.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0x4de35 0x230c5ee1 0x230cc2e5 0x22eb90d4 0x22d7937d 0x24db4880 0x24dbc036 0x24dbe984 0x24dbfb5d 0x2291858d 0x229691ea 0x2298cf1f 0x2ca25337 0x2cac3152 0x2cce73b0 0x2cce90d3 0x2c73e9fa 0x2c740903 0x1efbb1a0 0x1efc3f3d 0x1ee30484 0x3feaf 0x3ff5f 0x1e8daa44 2023-11-28T14:31:27.412 INFO:teuthology.orchestra.run.smithi055.stderr:kernel callstack: 2023-11-28T14:31:27.424 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:27,424 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-11-28T14:31:27.424 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:27,424 [shard 0] osd - get_early_config: set --smp 3 2023-11-28T14:31:27.941 INFO:teuthology.orchestra.run.smithi055.stderr:----------------------------------------------------- 2023-11-28T14:31:27.941 INFO:teuthology.orchestra.run.smithi055.stderr:Suppressions used: 2023-11-28T14:31:27.941 INFO:teuthology.orchestra.run.smithi055.stderr: count bytes template 2023-11-28T14:31:27.941 INFO:teuthology.orchestra.run.smithi055.stderr: 1 8 MallocExtension::Register 2023-11-28T14:31:27.942 INFO:teuthology.orchestra.run.smithi055.stderr:----------------------------------------------------- 2023-11-28T14:31:27.942 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-28T14:31:27.990 INFO:teuthology.orchestra.run.smithi055.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:31:27.991 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:27,991 seastar - Reactor backend: linux-aio 2023-11-28T14:31:28.009 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:28,009 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:31:28.009 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:28,009 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:31:28.009 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:28,009 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:31:28.016 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:28,016 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:31:28.019 INFO:teuthology.orchestra.run.smithi055.stderr:==80636==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:31:28.025 INFO:teuthology.orchestra.run.smithi055.stderr:==80636==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffcb51b000; bottom 0x7f0f98820000; size: 0x00f032cfb000 (1031644622848) 2023-11-28T14:31:28.025 INFO:teuthology.orchestra.run.smithi055.stderr:False positive error reports may follow 2023-11-28T14:31:28.025 INFO:teuthology.orchestra.run.smithi055.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:31:32.077 INFO:teuthology.orchestra.run.smithi055.stderr:----------------------------------------------------- 2023-11-28T14:31:32.077 INFO:teuthology.orchestra.run.smithi055.stderr:Suppressions used: 2023-11-28T14:31:32.077 INFO:teuthology.orchestra.run.smithi055.stderr: count bytes template 2023-11-28T14:31:32.078 INFO:teuthology.orchestra.run.smithi055.stderr: 1 8 MallocExtension::Register 2023-11-28T14:31:32.078 INFO:teuthology.orchestra.run.smithi055.stderr:----------------------------------------------------- 2023-11-28T14:31:32.078 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-28T14:31:32.078 INFO:teuthology.orchestra.run.smithi055.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 8639ce7e-f555-422d-b8f1-79137ef3334b 2023-11-28T14:31:32.116 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-28T14:31:32.147 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:31:32.699 INFO:teuthology.orchestra.run.smithi055.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:31:32.700 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:32,699 seastar - Reactor backend: linux-aio 2023-11-28T14:31:32.715 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:32,715 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:31:32.716 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:32,715 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:31:32.716 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:32,715 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:31:32.716 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:32,715 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:31:32.718 INFO:teuthology.orchestra.run.smithi055.stderr:==80820==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:31:32.724 INFO:teuthology.orchestra.run.smithi055.stderr:==80820==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe3a7e8000; bottom 0x7f65af35d000; size: 0x00988b48b000 (655171825664) 2023-11-28T14:31:32.724 INFO:teuthology.orchestra.run.smithi055.stderr:False positive error reports may follow 2023-11-28T14:31:32.724 INFO:teuthology.orchestra.run.smithi055.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:31:32.750 INFO:teuthology.orchestra.run.smithi055.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0xa111c 0xc384c 0xc161c 0x2c68d 0x2857c 0xb6323 0x1e97cc87 0x1e97db83 0x2038f271 0x230c573b 0x230c7615 0x230cda79 0x22eb90d4 0x22d7937d 0x24db4880 0x24dbc036 0x24dbe984 0x24dbfb5d 0x2291858d 0x229691ea 0x2298cf1f 0x2ca25337 0x2cac3152 0x2cce73b0 0x2cce90d3 0x2c73e9fa 0x2c740903 0x1efbb1a0 0x1efc3f3d 0x1ee30484 0x3feaf 0x3ff5f 0x1e8daa44 2023-11-28T14:31:32.750 INFO:teuthology.orchestra.run.smithi055.stderr:kernel callstack: 2023-11-28T14:31:32.762 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:32,761 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-11-28T14:31:32.762 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:32,761 [shard 0] osd - get_early_config: set --smp 3 2023-11-28T14:31:33.259 INFO:teuthology.orchestra.run.smithi055.stderr:----------------------------------------------------- 2023-11-28T14:31:33.259 INFO:teuthology.orchestra.run.smithi055.stderr:Suppressions used: 2023-11-28T14:31:33.260 INFO:teuthology.orchestra.run.smithi055.stderr: count bytes template 2023-11-28T14:31:33.260 INFO:teuthology.orchestra.run.smithi055.stderr: 1 8 MallocExtension::Register 2023-11-28T14:31:33.260 INFO:teuthology.orchestra.run.smithi055.stderr:----------------------------------------------------- 2023-11-28T14:31:33.260 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-28T14:31:33.316 INFO:teuthology.orchestra.run.smithi055.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:31:33.317 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:33,317 seastar - Reactor backend: linux-aio 2023-11-28T14:31:33.343 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:33,342 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:31:33.343 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:33,342 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:31:33.343 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:33,343 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:31:33.343 INFO:teuthology.orchestra.run.smithi055.stderr:INFO 2023-11-28 14:31:33,343 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:31:33.353 INFO:teuthology.orchestra.run.smithi055.stderr:==80818==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:31:33.359 INFO:teuthology.orchestra.run.smithi055.stderr:==80818==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe3a7e8000; bottom 0x7f65ad156000; size: 0x00988d692000 (655207505920) 2023-11-28T14:31:33.359 INFO:teuthology.orchestra.run.smithi055.stderr:False positive error reports may follow 2023-11-28T14:31:33.359 INFO:teuthology.orchestra.run.smithi055.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:31:37.399 INFO:teuthology.orchestra.run.smithi055.stderr:----------------------------------------------------- 2023-11-28T14:31:37.399 INFO:teuthology.orchestra.run.smithi055.stderr:Suppressions used: 2023-11-28T14:31:37.399 INFO:teuthology.orchestra.run.smithi055.stderr: count bytes template 2023-11-28T14:31:37.399 INFO:teuthology.orchestra.run.smithi055.stderr: 1 8 MallocExtension::Register 2023-11-28T14:31:37.399 INFO:teuthology.orchestra.run.smithi055.stderr:----------------------------------------------------- 2023-11-28T14:31:37.399 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-28T14:31:37.399 INFO:teuthology.orchestra.run.smithi055.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 8639ce7e-f555-422d-b8f1-79137ef3334b 2023-11-28T14:31:37.438 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-28T14:31:37.469 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-28T14:31:37.506 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-11-28T14:31:37.506 INFO:tasks.ceph:role: osd.2 2023-11-28T14:31:37.506 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2023-11-28T14:31:37.506 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-28T14:31:37.580 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T14:31:37.580 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T14:31:37.580 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T14:31:37.581 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T14:31:37.581 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T14:31:37.581 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2023-11-28T14:31:37.581 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T14:31:37.581 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T14:31:37.581 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T14:31:37.581 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T14:31:37.613 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2023-11-28T14:31:37.615 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2023-11-28T14:31:37.615 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2023-11-28T14:31:37.692 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-28T14:31:37.769 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-28T14:31:37.840 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-11-28T14:31:37.840 INFO:tasks.ceph:role: osd.3 2023-11-28T14:31:37.840 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2023-11-28T14:31:37.840 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-28T14:31:37.910 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T14:31:37.910 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T14:31:37.910 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T14:31:37.910 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T14:31:37.911 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T14:31:37.911 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2023-11-28T14:31:37.911 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T14:31:37.911 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T14:31:37.911 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T14:31:37.911 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T14:31:37.943 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2023-11-28T14:31:37.945 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2023-11-28T14:31:37.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2023-11-28T14:31:38.022 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-28T14:31:38.101 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:31:38.670 INFO:teuthology.orchestra.run.smithi060.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:31:38.670 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:38,670 seastar - Reactor backend: linux-aio 2023-11-28T14:31:38.691 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:38,691 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:31:38.691 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:38,691 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:31:38.692 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:38,691 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:31:38.692 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:38,691 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:31:38.694 INFO:teuthology.orchestra.run.smithi060.stderr:==80245==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:31:38.700 INFO:teuthology.orchestra.run.smithi060.stderr:==80245==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd686fe000; bottom 0x7f8f4892b000; size: 0x006e1fdd3000 (472980992000) 2023-11-28T14:31:38.700 INFO:teuthology.orchestra.run.smithi060.stderr:False positive error reports may follow 2023-11-28T14:31:38.700 INFO:teuthology.orchestra.run.smithi060.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:31:38.726 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0x230c5765 0x230c7615 0x230cda79 0x22eb90d4 0x22d7937d 0x24db4880 0x24dbc036 0x24dbe984 0x24dbfb5d 0x2291858d 0x229691ea 0x2298cf1f 0x2ca25337 0x2cac3152 0x2cce73b0 0x2cce90d3 0x2c73e9fa 0x2c740903 0x1efbb1a0 0x1efc3f3d 0x1ee30484 0x3feaf 0x3ff5f 0x1e8daa44 2023-11-28T14:31:38.726 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2023-11-28T14:31:38.739 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:38,738 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-11-28T14:31:38.739 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:38,738 [shard 0] osd - get_early_config: set --smp 3 2023-11-28T14:31:39.271 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2023-11-28T14:31:39.271 INFO:teuthology.orchestra.run.smithi060.stderr:Suppressions used: 2023-11-28T14:31:39.271 INFO:teuthology.orchestra.run.smithi060.stderr: count bytes template 2023-11-28T14:31:39.271 INFO:teuthology.orchestra.run.smithi060.stderr: 1 8 MallocExtension::Register 2023-11-28T14:31:39.272 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2023-11-28T14:31:39.272 INFO:teuthology.orchestra.run.smithi060.stderr: 2023-11-28T14:31:39.328 INFO:teuthology.orchestra.run.smithi060.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:31:39.329 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:39,329 seastar - Reactor backend: linux-aio 2023-11-28T14:31:39.347 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:39,347 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:31:39.347 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:39,347 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:31:39.347 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:39,347 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:31:39.354 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:39,354 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:31:39.357 INFO:teuthology.orchestra.run.smithi060.stderr:==80237==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:31:39.363 INFO:teuthology.orchestra.run.smithi060.stderr:==80237==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd686fe000; bottom 0x7f8f46724000; size: 0x006e21fda000 (473016672256) 2023-11-28T14:31:39.363 INFO:teuthology.orchestra.run.smithi060.stderr:False positive error reports may follow 2023-11-28T14:31:39.364 INFO:teuthology.orchestra.run.smithi060.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:31:39.449 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26619c5e 0x2661c2a7 0x2661e0b8 0x2662091d 0x26446cb1 0x1eeea284 0x1ee5ffaf 0x1ee63df0 0x2ce06e05 2023-11-28T14:31:39.449 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2023-11-28T14:31:43.436 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2023-11-28T14:31:43.437 INFO:teuthology.orchestra.run.smithi060.stderr:Suppressions used: 2023-11-28T14:31:43.437 INFO:teuthology.orchestra.run.smithi060.stderr: count bytes template 2023-11-28T14:31:43.437 INFO:teuthology.orchestra.run.smithi060.stderr: 1 8 MallocExtension::Register 2023-11-28T14:31:43.437 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2023-11-28T14:31:43.437 INFO:teuthology.orchestra.run.smithi060.stderr: 2023-11-28T14:31:43.437 INFO:teuthology.orchestra.run.smithi060.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 8639ce7e-f555-422d-b8f1-79137ef3334b 2023-11-28T14:31:43.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-28T14:31:43.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:31:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:31:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:44,058 seastar - Reactor backend: linux-aio 2023-11-28T14:31:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:44,074 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:31:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:44,074 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:31:44.077 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:44,074 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:31:44.077 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:44,074 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:31:44.077 INFO:teuthology.orchestra.run.smithi060.stderr:==80428==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:31:44.083 INFO:teuthology.orchestra.run.smithi060.stderr:==80428==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcb1a09000; bottom 0x7f4075b71000; size: 0x00bc3be98000 (808459010048) 2023-11-28T14:31:44.083 INFO:teuthology.orchestra.run.smithi060.stderr:False positive error reports may follow 2023-11-28T14:31:44.083 INFO:teuthology.orchestra.run.smithi060.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:31:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0x22f55165 0x22f7690c 0x230c707f 0x230c7615 0x230cda79 0x22eb90d4 0x22d7937d 0x24db4880 0x24dbc036 0x24dbe984 0x24dbfb5d 0x2291858d 0x229691ea 0x2298cf1f 0x2ca25337 0x2cac3152 0x2cce73b0 0x2cce90d3 0x2c73e9fa 0x2c740903 0x1efbb1a0 0x1efc3f3d 0x1ee30484 0x3feaf 0x3ff5f 0x1e8daa44 2023-11-28T14:31:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2023-11-28T14:31:44.121 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:44,121 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-11-28T14:31:44.121 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:44,121 [shard 0] osd - get_early_config: set --smp 3 2023-11-28T14:31:44.618 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2023-11-28T14:31:44.618 INFO:teuthology.orchestra.run.smithi060.stderr:Suppressions used: 2023-11-28T14:31:44.618 INFO:teuthology.orchestra.run.smithi060.stderr: count bytes template 2023-11-28T14:31:44.618 INFO:teuthology.orchestra.run.smithi060.stderr: 1 8 MallocExtension::Register 2023-11-28T14:31:44.618 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2023-11-28T14:31:44.618 INFO:teuthology.orchestra.run.smithi060.stderr: 2023-11-28T14:31:44.676 INFO:teuthology.orchestra.run.smithi060.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:31:44.676 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:44,676 seastar - Reactor backend: linux-aio 2023-11-28T14:31:44.694 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:44,694 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:31:44.694 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:44,694 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:31:44.695 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:44,694 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:31:44.702 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2023-11-28 14:31:44,701 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:31:44.704 INFO:teuthology.orchestra.run.smithi060.stderr:==80426==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:31:44.710 INFO:teuthology.orchestra.run.smithi060.stderr:==80426==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcb1a09000; bottom 0x7f407396a000; size: 0x00bc3e09f000 (808494690304) 2023-11-28T14:31:44.710 INFO:teuthology.orchestra.run.smithi060.stderr:False positive error reports may follow 2023-11-28T14:31:44.710 INFO:teuthology.orchestra.run.smithi060.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:31:44.795 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26619c5e 0x2661c2a7 0x2661e0b8 0x2662091d 0x26446cb1 0x1eeea284 0x1ee5ffaf 0x1ee63df0 0x2ce06e05 2023-11-28T14:31:44.809 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa6465b17 0xffffffffa6461447 0xffffffffa66000ea 2023-11-28T14:31:48.770 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2023-11-28T14:31:48.770 INFO:teuthology.orchestra.run.smithi060.stderr:Suppressions used: 2023-11-28T14:31:48.770 INFO:teuthology.orchestra.run.smithi060.stderr: count bytes template 2023-11-28T14:31:48.770 INFO:teuthology.orchestra.run.smithi060.stderr: 1 8 MallocExtension::Register 2023-11-28T14:31:48.770 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2023-11-28T14:31:48.770 INFO:teuthology.orchestra.run.smithi060.stderr: 2023-11-28T14:31:48.771 INFO:teuthology.orchestra.run.smithi060.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid 8639ce7e-f555-422d-b8f1-79137ef3334b 2023-11-28T14:31:48.809 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-28T14:31:48.840 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-28T14:31:48.840 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:48.840 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-28T14:31:48.877 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:48.877 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-28T14:31:48.951 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:31:48.951 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-28T14:31:48.982 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:31:48.982 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-28T14:31:49.054 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:31:49.054 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-28T14:31:49.124 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:49.124 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-28T14:31:49.143 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:31:49.143 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-11-28T14:31:49.180 INFO:tasks.ceph:Adding keys to all mons... 2023-11-28T14:31:49.180 DEBUG:teuthology.orchestra.run.smithi055:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-28T14:31:49.217 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.0] 2023-11-28T14:31:49.217 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQDA+WVl8+ehBBAAUdGSWREQEXj/vlAxTiZ3tQ== 2023-11-28T14:31:49.217 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.1] 2023-11-28T14:31:49.218 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQDF+WVlCNR6GBAAXnvYce3FWHyw+e8PbQfCCg== 2023-11-28T14:31:49.218 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr.x] 2023-11-28T14:31:49.218 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQC7+WVlTC3HGhAAg41HfjUIUWoz7zcvqlta3Q== 2023-11-28T14:31:49.218 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.2] 2023-11-28T14:31:49.218 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQDL+WVlTXjNGBAAgby7vFUM7sY4/ovrdlRYDA== 2023-11-28T14:31:49.218 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.3] 2023-11-28T14:31:49.218 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQDQ+WVlJ/tuLRAAjcDjosK+3ZBpnL/fCsvl3A== 2023-11-28T14:31:49.218 INFO:teuthology.orchestra.run.smithi055.stdout:[client.0] 2023-11-28T14:31:49.218 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQC7+WVlzbsWHhAArcGtOHDnAo3W828OUfFscA== 2023-11-28T14:31:49.218 INFO:teuthology.orchestra.run.smithi055.stdout:[client.1] 2023-11-28T14:31:49.218 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQC7+WVlaUViIhAA/oSubFMjsluX0vYgSjog+w== 2023-11-28T14:31:49.219 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T14:31:49.331 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T14:31:49.397 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T14:31:49.460 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T14:31:49.521 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T14:31:49.583 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T14:31:49.647 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T14:31:49.712 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-28T14:31:49.712 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-28T14:31:49.742 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T14:31:49.843 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:31:49.841+0000 7fe02c638980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:49.843 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:31:49.841+0000 7fe02c638980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:49.871 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-28T14:31:49.902 DEBUG:teuthology.orchestra.run.smithi055:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:31:49.957 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-28T14:31:49.957 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-28T14:31:49.957 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-28T14:31:49.999 INFO:tasks.ceph.mon.a:Started 2023-11-28T14:31:49.999 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-28T14:31:49.999 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-28T14:31:50.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-28T14:31:50.002 INFO:tasks.ceph.mgr.x:Started 2023-11-28T14:31:50.002 INFO:tasks.ceph:Setting crush tunables to default 2023-11-28T14:31:50.002 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd crush tunables default 2023-11-28T14:31:50.082 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:31:50.080+0000 7ff94ac1b980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:50.083 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:31:50.080+0000 7ff94ac1b980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:50.083 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:31:50.080+0000 7ff94ac1b980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:50.104 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:50.102+0000 7f5f4816af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:50.113 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:31:50.105+0000 7fde06911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:50.144 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:31:50.142+0000 7fde06911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:50.145 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:50.142+0000 7f5f4816af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:50.145 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:50.142+0000 7f5f4816af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:50.216 INFO:teuthology.orchestra.run.smithi055.stderr:adjusted tunables profile to default 2023-11-28T14:31:50.230 INFO:tasks.ceph:check_enable_crimson: True 2023-11-28T14:31:50.230 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-11-28T14:31:50.230 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-11-28T14:31:50.340 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:31:50.333+0000 7faa8c500640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:50.343 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:31:50.341+0000 7faa8c500640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:50.423 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:31:50.421+0000 7ff942bf1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-11-28T14:31:50.424 INFO:tasks.ceph.mon.a.smithi055.stderr:Please be aware that this feature is experimental, untested, 2023-11-28T14:31:50.424 INFO:tasks.ceph.mon.a.smithi055.stderr:unsupported, and may result in data corruption, data loss, 2023-11-28T14:31:50.424 INFO:tasks.ceph.mon.a.smithi055.stderr:and/or irreparable damage to your cluster. Do not use 2023-11-28T14:31:50.424 INFO:tasks.ceph.mon.a.smithi055.stderr:feature with important data. 2023-11-28T14:31:50.424 INFO:tasks.ceph.mon.a.smithi055.stderr: 2023-11-28T14:31:50.609 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:50.607+0000 7f5f4816af00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T14:31:50.663 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:50.661+0000 7f5f4816af00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T14:31:51.211 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:51.209+0000 7f5f4816af00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T14:31:51.235 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-28T14:31:51.235 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:51.235 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-28T14:31:51.266 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:51.264+0000 7f5f4816af00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T14:31:51.270 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-28T14:31:51.270 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-28T14:31:51.313 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:51.311+0000 7f5f4816af00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T14:31:51.344 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:31:51.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-28T14:31:51.370 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:51.368+0000 7f5f4816af00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T14:31:51.375 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-11-28T14:31:51.375 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-28T14:31:51.415 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:51.413+0000 7f5f4816af00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T14:31:51.446 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new dd11719f-98ab-4ed7-bd6f-76f9baae2b36 0 2023-11-28T14:31:51.524 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:51.522+0000 7f5f4816af00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T14:31:51.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-11-28T14:31:51.590+0000 7f10ec95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-11-28T14:31:51.598+0000 7f10ec95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:51.600 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:51.598+0000 7f5f4816af00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T14:31:51.658 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:51.656+0000 7f5f4816af00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T14:31:51.693 INFO:teuthology.orchestra.run.smithi060.stdout:0 2023-11-28T14:31:51.705 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new da6a0112-7fd5-4d4b-8e69-c03b74c92133 1 2023-11-28T14:31:51.714 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:51.712+0000 7f5f4816af00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T14:31:51.765 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:51.762+0000 7f5f4816af00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T14:31:51.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-11-28T14:31:51.817+0000 7f4ffcf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:51.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-11-28T14:31:51.824+0000 7f4ffcf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:51.908 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-11-28T14:31:51.919 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new 9fb8a5d0-ee28-4102-8e09-9dccf0c0e5b9 2 2023-11-28T14:31:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-11-28T14:31:52.022+0000 7f085671e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-11-28T14:31:52.031+0000 7f085671e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:52.113 INFO:teuthology.orchestra.run.smithi060.stdout:2 2023-11-28T14:31:52.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new 57090d75-bdf8-4514-b06c-8805db5e05ae 3 2023-11-28T14:31:52.179 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:52.177+0000 7f5f4816af00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T14:31:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-11-28T14:31:52.229+0000 7f693e15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:52.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-11-28T14:31:52.237+0000 7f693e15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:52.316 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:52.314+0000 7f5f4816af00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T14:31:52.319 INFO:teuthology.orchestra.run.smithi060.stdout:3 2023-11-28T14:31:52.330 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-11-28T14:31:52.371 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-28T14:31:52.371 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-11-28T14:31:52.413 INFO:tasks.ceph.osd.0:Started 2023-11-28T14:31:52.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-11-28T14:31:52.429 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:52.427+0000 7f5f4816af00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T14:31:52.452 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-28T14:31:52.452 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-11-28T14:31:52.477 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:52.475+0000 7f5f4816af00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T14:31:52.494 INFO:tasks.ceph.osd.1:Started 2023-11-28T14:31:52.494 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-11-28T14:31:52.530 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-28T14:31:52.530 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-11-28T14:31:52.572 INFO:tasks.ceph.osd.2:Started 2023-11-28T14:31:52.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2023-11-28T14:31:52.581 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:52.579+0000 7f5f4816af00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T14:31:52.607 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-28T14:31:52.607 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2023-11-28T14:31:52.649 INFO:tasks.ceph.osd.3:Started 2023-11-28T14:31:52.650 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T14:31:52.689 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:52.687+0000 7f5f4816af00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T14:31:52.779 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:31:52.773+0000 7f8f9126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:52.782 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:31:52.781+0000 7f8f9126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:52.852 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:31:52.852 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":7,"fsid":"8639ce7e-f555-422d-b8f1-79137ef3334b","created":"2023-11-28T14:31:50.125052+0000","modified":"2023-11-28T14:31:52.318268+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T14:31:52.318268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dd11719f-98ab-4ed7-bd6f-76f9baae2b36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"da6a0112-7fd5-4d4b-8e69-c03b74c92133","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9fb8a5d0-ee28-4102-8e09-9dccf0c0e5b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"57090d75-bdf8-4514-b06c-8805db5e05ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T14:31:52.859 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:52.857+0000 7f5f4816af00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T14:31:52.864 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-28T14:31:52.865 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-28T14:31:52.913 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:52.911+0000 7f5f4816af00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T14:31:53.080 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:53.078+0000 7f5f4816af00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T14:31:53.267 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T14:31:53.367 INFO:teuthology.misc.health.smithi055.stderr:2023-11-28T14:31:53.363+0000 7f19296ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:53.371 INFO:teuthology.misc.health.smithi055.stderr:2023-11-28T14:31:53.370+0000 7f19296ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:53.438 INFO:teuthology.misc.health.smithi055.stdout: 2023-11-28T14:31:53.439 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":7,"fsid":"8639ce7e-f555-422d-b8f1-79137ef3334b","created":"2023-11-28T14:31:50.125052+0000","modified":"2023-11-28T14:31:52.318268+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T14:31:52.318268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dd11719f-98ab-4ed7-bd6f-76f9baae2b36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"da6a0112-7fd5-4d4b-8e69-c03b74c92133","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9fb8a5d0-ee28-4102-8e09-9dccf0c0e5b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"57090d75-bdf8-4514-b06c-8805db5e05ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T14:31:53.449 DEBUG:teuthology.misc:0 of 4 OSDs are up 2023-11-28T14:31:53.969 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:53.967+0000 7f5f4816af00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T14:31:54.266 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:54.264+0000 7f5f4816af00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T14:31:54.329 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:54.327+0000 7f5f4816af00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T14:31:54.439 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:54.437+0000 7f5f4816af00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T14:31:55.460 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:31:55.458+0000 7f5ee7e3f640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-11-28T14:31:59.853 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T14:31:59.960 INFO:teuthology.misc.health.smithi055.stderr:2023-11-28T14:31:59.955+0000 7f7320f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:31:59.963 INFO:teuthology.misc.health.smithi055.stderr:2023-11-28T14:31:59.962+0000 7f7320f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:00.153 INFO:teuthology.misc.health.smithi055.stdout: 2023-11-28T14:32:00.252 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"8639ce7e-f555-422d-b8f1-79137ef3334b","created":"2023-11-28T14:31:50.125052+0000","modified":"2023-11-28T14:31:59.566481+0000","last_up_change":"2023-11-28T14:31:57.467051+0000","last_in_change":"2023-11-28T14:31:52.318268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T14:31:58.515285+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dd11719f-98ab-4ed7-bd6f-76f9baae2b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1538959351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6801","nonce":1538959351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1538959351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6805","nonce":1538959351}]},"public_addr":"172.21.15.55:6800/1538959351","cluster_addr":"172.21.15.55:6801/1538959351","heartbeat_back_addr":"172.21.15.55:6804/1538959351","heartbeat_front_addr":"172.21.15.55:6805/1538959351","state":["exists","up"]},{"osd":1,"uuid":"da6a0112-7fd5-4d4b-8e69-c03b74c92133","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":467134020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6803","nonce":467134020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":467134020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6807","nonce":467134020}]},"public_addr":"172.21.15.55:6802/467134020","cluster_addr":"172.21.15.55:6803/467134020","heartbeat_back_addr":"172.21.15.55:6806/467134020","heartbeat_front_addr":"172.21.15.55:6807/467134020","state":["exists","up"]},{"osd":2,"uuid":"9fb8a5d0-ee28-4102-8e09-9dccf0c0e5b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1135470082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1135470082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1135470082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1135470082}]},"public_addr":"172.21.15.60:6803/1135470082","cluster_addr":"172.21.15.60:6802/1135470082","heartbeat_back_addr":"172.21.15.60:6807/1135470082","heartbeat_front_addr":"172.21.15.60:6806/1135470082","state":["exists","up"]},{"osd":3,"uuid":"57090d75-bdf8-4514-b06c-8805db5e05ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6805","nonce":3231832409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3231832409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3231832409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":3231832409}]},"public_addr":"172.21.15.60:6805/3231832409","cluster_addr":"172.21.15.60:6804/3231832409","heartbeat_back_addr":"172.21.15.60:6808/3231832409","heartbeat_front_addr":"172.21.15.60:6809/3231832409","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T14:32:00.253 DEBUG:teuthology.misc:4 of 4 OSDs are up 2023-11-28T14:32:00.254 INFO:tasks.ceph:Creating RBD pool 2023-11-28T14:32:00.254 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-28T14:32:00.371 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:00.363+0000 7f156795e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:00.374 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:00.373+0000 7f156795e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:01.629 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'rbd' created 2023-11-28T14:32:01.645 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-28T14:32:01.778 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:01.769+0000 7fbea891e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:01.781 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:01.780+0000 7fbea891e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:02.642 INFO:teuthology.orchestra.run.smithi055.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-28T14:32:02.658 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-28T14:32:02.659 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-28T14:32:02.659 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-28T14:32:02.659 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-28T14:32:02.659 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-28T14:32:02.787 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:02.783+0000 7f461d71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:02.792 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:02.791+0000 7f461d71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:03.038 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1023645070},{"type":"v1","addr":"172.21.15.60:6801","nonce":1023645070}]},"active_addr":"172.21.15.60:6801/1023645070","active_change":"2023-11-28T14:31:54.452715+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":2259091115}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":2529827473}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":3074738728}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":1814203654}]}]} 2023-11-28T14:32:03.041 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-28T14:32:03.041 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-28T14:32:03.041 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T14:32:03.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.148+0000 7fb3b0f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.155 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.155+0000 7fb3b0f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.341 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:03.341 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":14,"fsid":"8639ce7e-f555-422d-b8f1-79137ef3334b","created":"2023-11-28T14:31:50.125052+0000","modified":"2023-11-28T14:32:02.640388+0000","last_up_change":"2023-11-28T14:31:57.467051+0000","last_in_change":"2023-11-28T14:31:52.318268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T14:31:58.515285+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T14:32:00.657972+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dd11719f-98ab-4ed7-bd6f-76f9baae2b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1538959351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6801","nonce":1538959351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1538959351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6805","nonce":1538959351}]},"public_addr":"172.21.15.55:6800/1538959351","cluster_addr":"172.21.15.55:6801/1538959351","heartbeat_back_addr":"172.21.15.55:6804/1538959351","heartbeat_front_addr":"172.21.15.55:6805/1538959351","state":["exists","up"]},{"osd":1,"uuid":"da6a0112-7fd5-4d4b-8e69-c03b74c92133","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":467134020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6803","nonce":467134020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":467134020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6807","nonce":467134020}]},"public_addr":"172.21.15.55:6802/467134020","cluster_addr":"172.21.15.55:6803/467134020","heartbeat_back_addr":"172.21.15.55:6806/467134020","heartbeat_front_addr":"172.21.15.55:6807/467134020","state":["exists","up"]},{"osd":2,"uuid":"9fb8a5d0-ee28-4102-8e09-9dccf0c0e5b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1135470082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1135470082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1135470082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1135470082}]},"public_addr":"172.21.15.60:6803/1135470082","cluster_addr":"172.21.15.60:6802/1135470082","heartbeat_back_addr":"172.21.15.60:6807/1135470082","heartbeat_front_addr":"172.21.15.60:6806/1135470082","state":["exists","up"]},{"osd":3,"uuid":"57090d75-bdf8-4514-b06c-8805db5e05ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6805","nonce":3231832409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3231832409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3231832409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":3231832409}]},"public_addr":"172.21.15.60:6805/3231832409","cluster_addr":"172.21.15.60:6804/3231832409","heartbeat_back_addr":"172.21.15.60:6808/3231832409","heartbeat_front_addr":"172.21.15.60:6809/3231832409","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T14:32:03.353 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-28T14:32:03.353 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T14:32:03.465 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.461+0000 7fba4e46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.468 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.467+0000 7fba4e46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.659 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:03.659 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":14,"fsid":"8639ce7e-f555-422d-b8f1-79137ef3334b","created":"2023-11-28T14:31:50.125052+0000","modified":"2023-11-28T14:32:02.640388+0000","last_up_change":"2023-11-28T14:31:57.467051+0000","last_in_change":"2023-11-28T14:31:52.318268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T14:31:58.515285+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T14:32:00.657972+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dd11719f-98ab-4ed7-bd6f-76f9baae2b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1538959351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6801","nonce":1538959351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1538959351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6805","nonce":1538959351}]},"public_addr":"172.21.15.55:6800/1538959351","cluster_addr":"172.21.15.55:6801/1538959351","heartbeat_back_addr":"172.21.15.55:6804/1538959351","heartbeat_front_addr":"172.21.15.55:6805/1538959351","state":["exists","up"]},{"osd":1,"uuid":"da6a0112-7fd5-4d4b-8e69-c03b74c92133","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":467134020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6803","nonce":467134020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":467134020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6807","nonce":467134020}]},"public_addr":"172.21.15.55:6802/467134020","cluster_addr":"172.21.15.55:6803/467134020","heartbeat_back_addr":"172.21.15.55:6806/467134020","heartbeat_front_addr":"172.21.15.55:6807/467134020","state":["exists","up"]},{"osd":2,"uuid":"9fb8a5d0-ee28-4102-8e09-9dccf0c0e5b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1135470082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1135470082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1135470082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1135470082}]},"public_addr":"172.21.15.60:6803/1135470082","cluster_addr":"172.21.15.60:6802/1135470082","heartbeat_back_addr":"172.21.15.60:6807/1135470082","heartbeat_front_addr":"172.21.15.60:6806/1135470082","state":["exists","up"]},{"osd":3,"uuid":"57090d75-bdf8-4514-b06c-8805db5e05ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6805","nonce":3231832409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3231832409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3231832409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":3231832409}]},"public_addr":"172.21.15.60:6805/3231832409","cluster_addr":"172.21.15.60:6804/3231832409","heartbeat_back_addr":"172.21.15.60:6808/3231832409","heartbeat_front_addr":"172.21.15.60:6809/3231832409","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T14:32:03.673 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-28T14:32:03.675 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-28T14:32:03.675 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-28T14:32:03.676 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-28T14:32:03.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.795+0000 7f17e079e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.795+0000 7efe4d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.795+0000 7f65e2660640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.795+0000 7ffb7b42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.804 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.803+0000 7f65e2660640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.804 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.803+0000 7efe4d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.805 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.803+0000 7f17e079e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.805 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.804+0000 7ffb7b42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.833 INFO:teuthology.orchestra.run.smithi055.stdout:38654705670 2023-11-28T14:32:03.834 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-28T14:32:03.835 INFO:teuthology.orchestra.run.smithi055.stdout:38654705670 2023-11-28T14:32:03.835 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-28T14:32:03.840 INFO:teuthology.orchestra.run.smithi055.stdout:38654705670 2023-11-28T14:32:03.840 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-28T14:32:03.841 INFO:teuthology.orchestra.run.smithi055.stdout:38654705670 2023-11-28T14:32:03.841 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-28T14:32:03.950 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.945+0000 7f7237a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.954 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.953+0000 7f7237a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:03.999 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.994+0000 7f267c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:04.000 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:03.994+0000 7f5be7f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:04.004 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:04.003+0000 7f267c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:04.005 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:04.004+0000 7f5be7f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:04.011 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:04.004+0000 7ff2ed955640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:04.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:04.020+0000 7ff2ed955640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:04.161 INFO:teuthology.orchestra.run.smithi055.stdout:38654705670 2023-11-28T14:32:04.174 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.3 2023-11-28T14:32:04.175 DEBUG:teuthology.parallel:result is None 2023-11-28T14:32:04.201 INFO:teuthology.orchestra.run.smithi055.stdout:38654705670 2023-11-28T14:32:04.212 INFO:teuthology.orchestra.run.smithi055.stdout:38654705670 2023-11-28T14:32:04.213 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.2 2023-11-28T14:32:04.214 DEBUG:teuthology.parallel:result is None 2023-11-28T14:32:04.225 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.0 2023-11-28T14:32:04.225 DEBUG:teuthology.parallel:result is None 2023-11-28T14:32:04.226 INFO:teuthology.orchestra.run.smithi055.stdout:38654705670 2023-11-28T14:32:04.239 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.1 2023-11-28T14:32:04.239 DEBUG:teuthology.parallel:result is None 2023-11-28T14:32:04.239 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-28T14:32:04.239 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:32:04.391 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:04.386+0000 7fc12c02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:04.394 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:04.393+0000 7fc12c02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:04.578 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:04.579 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:32:04.590 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-11-28T14:32:04.457853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107744,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379616,"statfs":{"total":386547056640,"available":386436726784,"internally_reserved":0,"allocated":430080,"data_stored":1858476,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.790656"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.775877+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:32:02.775877+0000","last_peered":"2023-11-28T14:32:02.775877+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:32:02.775877+0000","last_undegraded":"2023-11-28T14:32:02.775877+0000","last_fullsized":"2023-11-28T14:32:02.775877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-28T14:32:03.677692+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:32:03.677692+0000","last_peered":"2023-11-28T14:32:03.677692+0000","last_clean":"2023-11-28T14:32:03.677692+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:32:03.677692+0000","last_undegraded":"2023-11-28T14:32:03.677692+0000","last_fullsized":"2023-11-28T14:32:03.677692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.734692+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:32:02.734692+0000","last_peered":"2023-11-28T14:32:02.734692+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:32:02.734692+0000","last_undegraded":"2023-11-28T14:32:02.734692+0000","last_fullsized":"2023-11-28T14:32:02.734692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-28T14:32:03.678346+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:32:03.678346+0000","last_peered":"2023-11-28T14:32:03.678346+0000","last_clean":"2023-11-28T14:32:03.678346+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:32:03.678346+0000","last_undegraded":"2023-11-28T14:32:03.678346+0000","last_fullsized":"2023-11-28T14:32:03.678346+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.757066+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:32:02.757066+0000","last_peered":"2023-11-28T14:32:02.757066+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:32:02.757066+0000","last_undegraded":"2023-11-28T14:32:02.757066+0000","last_fullsized":"2023-11-28T14:32:02.757066+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.741833+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:32:02.741833+0000","last_peered":"2023-11-28T14:32:02.741833+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:32:02.741833+0000","last_undegraded":"2023-11-28T14:32:02.741833+0000","last_fullsized":"2023-11-28T14:32:02.741833+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.770616+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:32:02.770616+0000","last_peered":"2023-11-28T14:32:02.770616+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:32:02.770616+0000","last_undegraded":"2023-11-28T14:32:02.770616+0000","last_fullsized":"2023-11-28T14:32:02.770616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.758730+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:32:02.758730+0000","last_peered":"2023-11-28T14:32:02.758730+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:32:02.758730+0000","last_undegraded":"2023-11-28T14:32:02.758730+0000","last_fullsized":"2023-11-28T14:32:02.758730+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.649369+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:32:02.649369+0000","last_peered":"2023-11-28T14:32:02.649369+0000","last_clean":"2023-11-28T14:32:02.649369+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:32:02.649369+0000","last_undegraded":"2023-11-28T14:32:02.649369+0000","last_fullsized":"2023-11-28T14:32:02.649369+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26956,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344884,"statfs":{"total":96636764160,"available":96609161216,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26884,"kb_used_data":60,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344956,"statfs":{"total":96636764160,"available":96609234944,"internally_reserved":0,"allocated":61440,"data_stored":21843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:32:04.591 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:32:04.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:04.698+0000 7fc52646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:04.705 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:04.704+0000 7fc52646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:04.888 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:04.889 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:32:04.900 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-11-28T14:32:04.457853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107744,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379616,"statfs":{"total":386547056640,"available":386436726784,"internally_reserved":0,"allocated":430080,"data_stored":1858476,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.790656"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.775877+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:32:02.775877+0000","last_peered":"2023-11-28T14:32:02.775877+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:32:02.775877+0000","last_undegraded":"2023-11-28T14:32:02.775877+0000","last_fullsized":"2023-11-28T14:32:02.775877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-28T14:32:03.677692+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:32:03.677692+0000","last_peered":"2023-11-28T14:32:03.677692+0000","last_clean":"2023-11-28T14:32:03.677692+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:32:03.677692+0000","last_undegraded":"2023-11-28T14:32:03.677692+0000","last_fullsized":"2023-11-28T14:32:03.677692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.734692+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:32:02.734692+0000","last_peered":"2023-11-28T14:32:02.734692+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:32:02.734692+0000","last_undegraded":"2023-11-28T14:32:02.734692+0000","last_fullsized":"2023-11-28T14:32:02.734692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-28T14:32:03.678346+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:32:03.678346+0000","last_peered":"2023-11-28T14:32:03.678346+0000","last_clean":"2023-11-28T14:32:03.678346+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:32:03.678346+0000","last_undegraded":"2023-11-28T14:32:03.678346+0000","last_fullsized":"2023-11-28T14:32:03.678346+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.757066+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:32:02.757066+0000","last_peered":"2023-11-28T14:32:02.757066+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:32:02.757066+0000","last_undegraded":"2023-11-28T14:32:02.757066+0000","last_fullsized":"2023-11-28T14:32:02.757066+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.741833+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:32:02.741833+0000","last_peered":"2023-11-28T14:32:02.741833+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:32:02.741833+0000","last_undegraded":"2023-11-28T14:32:02.741833+0000","last_fullsized":"2023-11-28T14:32:02.741833+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.770616+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:32:02.770616+0000","last_peered":"2023-11-28T14:32:02.770616+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:32:02.770616+0000","last_undegraded":"2023-11-28T14:32:02.770616+0000","last_fullsized":"2023-11-28T14:32:02.770616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.758730+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:32:02.758730+0000","last_peered":"2023-11-28T14:32:02.758730+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:32:02.758730+0000","last_undegraded":"2023-11-28T14:32:02.758730+0000","last_fullsized":"2023-11-28T14:32:02.758730+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.649369+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:32:02.649369+0000","last_peered":"2023-11-28T14:32:02.649369+0000","last_clean":"2023-11-28T14:32:02.649369+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:32:02.649369+0000","last_undegraded":"2023-11-28T14:32:02.649369+0000","last_fullsized":"2023-11-28T14:32:02.649369+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26956,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344884,"statfs":{"total":96636764160,"available":96609161216,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26884,"kb_used_data":60,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344956,"statfs":{"total":96636764160,"available":96609234944,"internally_reserved":0,"allocated":61440,"data_stored":21843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:32:04.902 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-28T14:32:04.902 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-28T14:32:04.902 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-28T14:32:04.902 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T14:32:05.013 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:05.009+0000 7f8aca95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:05.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:05.015+0000 7f8aca95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:05.237 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:05.237 INFO:teuthology.orchestra.run.smithi055.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-28T14:32:05.249 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-28T14:32:05.249 INFO:teuthology.run_tasks:Running task thrashosds... 2023-11-28T14:32:05.260 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2023-11-28T14:32:05.261 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'b9b6c42eed672399c3d01aa72f6cda1a665a3c35'}} 2023-11-28T14:32:05.261 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2023-11-28T14:32:05.261 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2023-11-28T14:32:05.375 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:05.368+0000 7fa514304640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:05.378 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:05.378+0000 7fa514304640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:05.583 INFO:tasks.thrashosds:Beginning thrashosds... 2023-11-28T14:32:05.583 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-28T14:32:05.583 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:32:05.695 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:05.691+0000 7fd58b31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:05.698 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:05.697+0000 7fd58b31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:05.888 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:32:05.899 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-11-28T14:32:04.457853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107744,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379616,"statfs":{"total":386547056640,"available":386436726784,"internally_reserved":0,"allocated":430080,"data_stored":1858476,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.790656"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.775877+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:32:02.775877+0000","last_peered":"2023-11-28T14:32:02.775877+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:32:02.775877+0000","last_undegraded":"2023-11-28T14:32:02.775877+0000","last_fullsized":"2023-11-28T14:32:02.775877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-28T14:32:03.677692+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:32:03.677692+0000","last_peered":"2023-11-28T14:32:03.677692+0000","last_clean":"2023-11-28T14:32:03.677692+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:32:03.677692+0000","last_undegraded":"2023-11-28T14:32:03.677692+0000","last_fullsized":"2023-11-28T14:32:03.677692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.734692+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:32:02.734692+0000","last_peered":"2023-11-28T14:32:02.734692+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:32:02.734692+0000","last_undegraded":"2023-11-28T14:32:02.734692+0000","last_fullsized":"2023-11-28T14:32:02.734692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-28T14:32:03.678346+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:32:03.678346+0000","last_peered":"2023-11-28T14:32:03.678346+0000","last_clean":"2023-11-28T14:32:03.678346+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:32:03.678346+0000","last_undegraded":"2023-11-28T14:32:03.678346+0000","last_fullsized":"2023-11-28T14:32:03.678346+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.757066+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:32:02.757066+0000","last_peered":"2023-11-28T14:32:02.757066+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:32:02.757066+0000","last_undegraded":"2023-11-28T14:32:02.757066+0000","last_fullsized":"2023-11-28T14:32:02.757066+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.741833+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:32:02.741833+0000","last_peered":"2023-11-28T14:32:02.741833+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:32:02.741833+0000","last_undegraded":"2023-11-28T14:32:02.741833+0000","last_fullsized":"2023-11-28T14:32:02.741833+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.770616+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:32:02.770616+0000","last_peered":"2023-11-28T14:32:02.770616+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:32:02.770616+0000","last_undegraded":"2023-11-28T14:32:02.770616+0000","last_fullsized":"2023-11-28T14:32:02.770616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.758730+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:32:02.758730+0000","last_peered":"2023-11-28T14:32:02.758730+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:32:02.758730+0000","last_undegraded":"2023-11-28T14:32:02.758730+0000","last_fullsized":"2023-11-28T14:32:02.758730+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.649369+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:32:02.649369+0000","last_peered":"2023-11-28T14:32:02.649369+0000","last_clean":"2023-11-28T14:32:02.649369+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:32:02.649369+0000","last_undegraded":"2023-11-28T14:32:02.649369+0000","last_fullsized":"2023-11-28T14:32:02.649369+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26956,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344884,"statfs":{"total":96636764160,"available":96609161216,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26884,"kb_used_data":60,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344956,"statfs":{"total":96636764160,"available":96609234944,"internally_reserved":0,"allocated":61440,"data_stored":21843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:32:05.900 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:32:06.012 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:06.008+0000 7f645ce24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:06.015 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:06.014+0000 7f645ce24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:06.200 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:06.200 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:32:06.211 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-11-28T14:32:04.457853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107744,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379616,"statfs":{"total":386547056640,"available":386436726784,"internally_reserved":0,"allocated":430080,"data_stored":1858476,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.790656"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.775877+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:32:02.775877+0000","last_peered":"2023-11-28T14:32:02.775877+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:32:02.775877+0000","last_undegraded":"2023-11-28T14:32:02.775877+0000","last_fullsized":"2023-11-28T14:32:02.775877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-28T14:32:03.677692+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:32:03.677692+0000","last_peered":"2023-11-28T14:32:03.677692+0000","last_clean":"2023-11-28T14:32:03.677692+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:32:03.677692+0000","last_undegraded":"2023-11-28T14:32:03.677692+0000","last_fullsized":"2023-11-28T14:32:03.677692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.734692+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:32:02.734692+0000","last_peered":"2023-11-28T14:32:02.734692+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:32:02.734692+0000","last_undegraded":"2023-11-28T14:32:02.734692+0000","last_fullsized":"2023-11-28T14:32:02.734692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-28T14:32:03.678346+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:32:03.678346+0000","last_peered":"2023-11-28T14:32:03.678346+0000","last_clean":"2023-11-28T14:32:03.678346+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:32:03.678346+0000","last_undegraded":"2023-11-28T14:32:03.678346+0000","last_fullsized":"2023-11-28T14:32:03.678346+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.757066+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:32:02.757066+0000","last_peered":"2023-11-28T14:32:02.757066+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:32:02.757066+0000","last_undegraded":"2023-11-28T14:32:02.757066+0000","last_fullsized":"2023-11-28T14:32:02.757066+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.741833+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:32:02.741833+0000","last_peered":"2023-11-28T14:32:02.741833+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:32:02.741833+0000","last_undegraded":"2023-11-28T14:32:02.741833+0000","last_fullsized":"2023-11-28T14:32:02.741833+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.770616+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:32:02.770616+0000","last_peered":"2023-11-28T14:32:02.770616+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:32:02.770616+0000","last_undegraded":"2023-11-28T14:32:02.770616+0000","last_fullsized":"2023-11-28T14:32:02.770616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.758730+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:32:02.758730+0000","last_peered":"2023-11-28T14:32:02.758730+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:32:02.758730+0000","last_undegraded":"2023-11-28T14:32:02.758730+0000","last_fullsized":"2023-11-28T14:32:02.758730+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.649369+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:32:02.649369+0000","last_peered":"2023-11-28T14:32:02.649369+0000","last_clean":"2023-11-28T14:32:02.649369+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:32:02.649369+0000","last_undegraded":"2023-11-28T14:32:02.649369+0000","last_fullsized":"2023-11-28T14:32:02.649369+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26956,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344884,"statfs":{"total":96636764160,"available":96609161216,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26884,"kb_used_data":60,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344956,"statfs":{"total":96636764160,"available":96609234944,"internally_reserved":0,"allocated":61440,"data_stored":21843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:32:06.212 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-28T14:32:06.212 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2023-11-28T14:32:06.324 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:06.319+0000 7f98a12f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:06.327 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:06.326+0000 7f98a12f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:06.515 INFO:teuthology.orchestra.run.smithi055.stdout:epoch 15 2023-11-28T14:32:06.515 INFO:teuthology.orchestra.run.smithi055.stdout:fsid 8639ce7e-f555-422d-b8f1-79137ef3334b 2023-11-28T14:32:06.516 INFO:teuthology.orchestra.run.smithi055.stdout:created 2023-11-28T14:31:50.125052+0000 2023-11-28T14:32:06.516 INFO:teuthology.orchestra.run.smithi055.stdout:modified 2023-11-28T14:32:03.660677+0000 2023-11-28T14:32:06.516 INFO:teuthology.orchestra.run.smithi055.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-11-28T14:32:06.516 INFO:teuthology.orchestra.run.smithi055.stdout:crush_version 3 2023-11-28T14:32:06.516 INFO:teuthology.orchestra.run.smithi055.stdout:full_ratio 0.95 2023-11-28T14:32:06.516 INFO:teuthology.orchestra.run.smithi055.stdout:backfillfull_ratio 0.9 2023-11-28T14:32:06.516 INFO:teuthology.orchestra.run.smithi055.stdout:nearfull_ratio 0.85 2023-11-28T14:32:06.517 INFO:teuthology.orchestra.run.smithi055.stdout:require_min_compat_client luminous 2023-11-28T14:32:06.517 INFO:teuthology.orchestra.run.smithi055.stdout:min_compat_client jewel 2023-11-28T14:32:06.517 INFO:teuthology.orchestra.run.smithi055.stdout:require_osd_release reef 2023-11-28T14:32:06.517 INFO:teuthology.orchestra.run.smithi055.stdout:stretch_mode_enabled false 2023-11-28T14:32:06.517 INFO:teuthology.orchestra.run.smithi055.stdout:allow_crimson=true 2023-11-28T14:32:06.518 INFO:teuthology.orchestra.run.smithi055.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 12 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2023-11-28T14:32:06.518 INFO:teuthology.orchestra.run.smithi055.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 15 flags hashpspool,nopgchange,crimson stripe_width 0 application rbd read_balance_score 1.50 2023-11-28T14:32:06.518 INFO:teuthology.orchestra.run.smithi055.stdout:max_osd 4 2023-11-28T14:32:06.518 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.55:6800/1538959351 v2:172.21.15.55:6801/1538959351 exists,up dd11719f-98ab-4ed7-bd6f-76f9baae2b36 2023-11-28T14:32:06.518 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.55:6802/467134020 v2:172.21.15.55:6803/467134020 exists,up da6a0112-7fd5-4d4b-8e69-c03b74c92133 2023-11-28T14:32:06.518 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.60:6803/1135470082 v2:172.21.15.60:6802/1135470082 exists,up 9fb8a5d0-ee28-4102-8e09-9dccf0c0e5b9 2023-11-28T14:32:06.519 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.60:6805/3231832409 v2:172.21.15.60:6804/3231832409 exists,up 57090d75-bdf8-4514-b06c-8805db5e05ae 2023-11-28T14:32:06.528 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.55:6800/1538959351 v2:172.21.15.55:6801/1538959351 exists,up dd11719f-98ab-4ed7-bd6f-76f9baae2b36', 'osd.1 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.55:6802/467134020 v2:172.21.15.55:6803/467134020 exists,up da6a0112-7fd5-4d4b-8e69-c03b74c92133', 'osd.2 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.60:6803/1135470082 v2:172.21.15.60:6802/1135470082 exists,up 9fb8a5d0-ee28-4102-8e09-9dccf0c0e5b9', 'osd.3 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.60:6805/3231832409 v2:172.21.15.60:6804/3231832409 exists,up 57090d75-bdf8-4514-b06c-8805db5e05ae'] 2023-11-28T14:32:06.528 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2023-11-28T14:32:06.642 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2023-11-28T14:32:06.642 INFO:teuthology.orchestra.run.smithi055.stdout: "name": "mon.a", 2023-11-28T14:32:06.642 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster": "ceph", 2023-11-28T14:32:06.642 INFO:teuthology.orchestra.run.smithi055.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2023-11-28T14:32:06.643 INFO:teuthology.orchestra.run.smithi055.stdout: "admin_socket_mode": "", 2023-11-28T14:32:06.643 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2023-11-28T14:32:06.643 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_client_required": "cephx, none", 2023-11-28T14:32:06.643 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_cluster_required": "cephx", 2023-11-28T14:32:06.643 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_debug": "true", 2023-11-28T14:32:06.643 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2023-11-28T14:32:06.643 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_mon_ticket_ttl": "660.000000", 2023-11-28T14:32:06.644 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_service_required": "cephx", 2023-11-28T14:32:06.644 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_service_ticket_ttl": "240.000000", 2023-11-28T14:32:06.644 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_supported": "cephx", 2023-11-28T14:32:06.644 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_aio": "true", 2023-11-28T14:32:06.644 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_aio_max_queue_depth": "1024", 2023-11-28T14:32:06.644 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_aio_poll_ms": "250", 2023-11-28T14:32:06.645 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_aio_reap_max": "16", 2023-11-28T14:32:06.645 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_async_discard": "false", 2023-11-28T14:32:06.645 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_block_size": "4096", 2023-11-28T14:32:06.645 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_debug_aio": "false", 2023-11-28T14:32:06.645 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_debug_aio_log_age": "5.000000", 2023-11-28T14:32:06.646 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2023-11-28T14:32:06.646 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_debug_inflight_ios": "false", 2023-11-28T14:32:06.646 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_enable_discard": "false", 2023-11-28T14:32:06.646 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_flock_retry": "3", 2023-11-28T14:32:06.646 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_flock_retry_interval": "0.100000", 2023-11-28T14:32:06.646 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_inject_crash": "0", 2023-11-28T14:32:06.647 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_inject_crash_flush_delay": "2", 2023-11-28T14:32:06.647 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_ioring": "false", 2023-11-28T14:32:06.647 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_ioring_hipri": "false", 2023-11-28T14:32:06.647 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_ioring_sqthread_poll": "false", 2023-11-28T14:32:06.647 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_nvme_unbind_from_kernel": "false", 2023-11-28T14:32:06.647 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_read_buffer_alignment": "4096", 2023-11-28T14:32:06.648 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_read_preallocated_huge_buffers": "", 2023-11-28T14:32:06.648 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_type": "", 2023-11-28T14:32:06.648 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_alloc_size": "1048576", 2023-11-28T14:32:06.648 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_allocator": "hybrid", 2023-11-28T14:32:06.648 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_buffered_io": "true", 2023-11-28T14:32:06.648 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_check_for_zeros": "false", 2023-11-28T14:32:06.649 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_check_volume_selector_often": "false", 2023-11-28T14:32:06.649 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_check_volume_selector_on_umount": "false", 2023-11-28T14:32:06.649 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_compact_log_sync": "false", 2023-11-28T14:32:06.649 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2023-11-28T14:32:06.649 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2023-11-28T14:32:06.649 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_log_compact_min_size": "16777216", 2023-11-28T14:32:06.650 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_log_replay_check_allocations": "true", 2023-11-28T14:32:06.650 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_max_log_runway": "4194304", 2023-11-28T14:32:06.650 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_max_prefetch": "1048576", 2023-11-28T14:32:06.650 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_min_flush_size": "524288", 2023-11-28T14:32:06.650 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_min_log_runway": "1048576", 2023-11-28T14:32:06.651 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_replay_recovery": "false", 2023-11-28T14:32:06.651 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_replay_recovery_disable_compact": "false", 2023-11-28T14:32:06.651 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_shared_alloc_size": "65536", 2023-11-28T14:32:06.651 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_sync_write": "false", 2023-11-28T14:32:06.652 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2023-11-28T14:32:06.652 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2023-11-28T14:32:06.652 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2023-11-28T14:32:06.652 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_allocation_from_file": "true", 2023-11-28T14:32:06.653 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_allocator": "hybrid", 2023-11-28T14:32:06.653 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2023-11-28T14:32:06.653 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2023-11-28T14:32:06.653 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2023-11-28T14:32:06.653 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2023-11-28T14:32:06.654 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2023-11-28T14:32:06.654 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_bitmapallocator_span_size": "1024", 2023-11-28T14:32:06.654 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_blobid_prealloc": "10240", 2023-11-28T14:32:06.654 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_create": "true", 2023-11-28T14:32:06.654 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_db_create": "false", 2023-11-28T14:32:06.654 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_db_path": "", 2023-11-28T14:32:06.655 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_db_size": "0", 2023-11-28T14:32:06.655 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_path": "", 2023-11-28T14:32:06.655 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_preallocate_file": "false", 2023-11-28T14:32:06.655 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_size": "107374182400", 2023-11-28T14:32:06.655 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_wal_create": "false", 2023-11-28T14:32:06.655 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_wal_path": "", 2023-11-28T14:32:06.655 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_wal_size": "100663296", 2023-11-28T14:32:06.655 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_bluefs": "true", 2023-11-28T14:32:06.656 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2023-11-28T14:32:06.656 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_bluefs_env_mirror": "false", 2023-11-28T14:32:06.656 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_bluefs_max_free": "10737418240", 2023-11-28T14:32:06.656 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2023-11-28T14:32:06.656 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2023-11-28T14:32:06.656 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2023-11-28T14:32:06.656 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2023-11-28T14:32:06.657 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2023-11-28T14:32:06.657 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_autotune": "true", 2023-11-28T14:32:06.657 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_autotune_interval": "5.000000", 2023-11-28T14:32:06.657 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2023-11-28T14:32:06.658 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_kv_ratio": "0.450000", 2023-11-28T14:32:06.658 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_meta_ratio": "0.450000", 2023-11-28T14:32:06.658 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_size": "0", 2023-11-28T14:32:06.658 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_size_hdd": "1073741824", 2023-11-28T14:32:06.658 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_size_ssd": "3221225472", 2023-11-28T14:32:06.658 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_trim_interval": "0.050000", 2023-11-28T14:32:06.659 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2023-11-28T14:32:06.659 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_type": "2q", 2023-11-28T14:32:06.659 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2023-11-28T14:32:06.659 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_clone_cow": "true", 2023-11-28T14:32:06.659 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_algorithm": "snappy", 2023-11-28T14:32:06.659 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_max_blob_size": "0", 2023-11-28T14:32:06.659 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2023-11-28T14:32:06.659 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2023-11-28T14:32:06.659 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_min_blob_size": "0", 2023-11-28T14:32:06.659 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2023-11-28T14:32:06.659 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2023-11-28T14:32:06.659 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_mode": "none", 2023-11-28T14:32:06.660 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_required_ratio": "0.875000", 2023-11-28T14:32:06.660 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_csum_type": "crc32c", 2023-11-28T14:32:06.660 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_enforce_settings": "default", 2023-11-28T14:32:06.660 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_freelist": "false", 2023-11-28T14:32:06.660 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_fsck_abort": "false", 2023-11-28T14:32:06.660 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2023-11-28T14:32:06.660 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2023-11-28T14:32:06.660 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_inject_read_err": "false", 2023-11-28T14:32:06.660 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_legacy_omap": "false", 2023-11-28T14:32:06.660 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_no_reuse_blocks": "false", 2023-11-28T14:32:06.660 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_omit_block_device_write": "false", 2023-11-28T14:32:06.660 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_omit_kv_commit": "false", 2023-11-28T14:32:06.660 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2023-11-28T14:32:06.661 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_prefill": "0.000000", 2023-11-28T14:32:06.661 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_prefragment_max": "1048576", 2023-11-28T14:32:06.661 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_random_read_err": "0.000000", 2023-11-28T14:32:06.661 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2023-11-28T14:32:06.661 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_small_allocations": "0", 2023-11-28T14:32:06.661 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2023-11-28T14:32:06.661 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_default_buffered_read": "true", 2023-11-28T14:32:06.661 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_default_buffered_write": "false", 2023-11-28T14:32:06.661 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_deferred_batch_ops": "0", 2023-11-28T14:32:06.661 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2023-11-28T14:32:06.661 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2023-11-28T14:32:06.661 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_elastic_shared_blobs": "true", 2023-11-28T14:32:06.662 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2023-11-28T14:32:06.662 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_map_shard_max_size": "1200", 2023-11-28T14:32:06.662 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_map_shard_min_size": "150", 2023-11-28T14:32:06.662 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_map_shard_target_size": "500", 2023-11-28T14:32:06.662 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2023-11-28T14:32:06.662 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fail_eio": "false", 2023-11-28T14:32:06.662 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_freelist_blocks_per_key": "128", 2023-11-28T14:32:06.662 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2023-11-28T14:32:06.663 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2023-11-28T14:32:06.663 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2023-11-28T14:32:06.663 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_on_mkfs": "true", 2023-11-28T14:32:06.663 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2023-11-28T14:32:06.663 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_on_mount": "false", 2023-11-28T14:32:06.663 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_on_mount_deep": "false", 2023-11-28T14:32:06.663 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_on_umount": "false", 2023-11-28T14:32:06.663 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_on_umount_deep": "false", 2023-11-28T14:32:06.663 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2023-11-28T14:32:06.663 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_quick_fix_threads": "2", 2023-11-28T14:32:06.664 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2023-11-28T14:32:06.664 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2023-11-28T14:32:06.664 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_gc_enable_blob_threshold": "0", 2023-11-28T14:32:06.664 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_gc_enable_total_threshold": "0", 2023-11-28T14:32:06.664 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2023-11-28T14:32:06.664 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_ignore_data_csum": "false", 2023-11-28T14:32:06.664 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2023-11-28T14:32:06.664 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_kvbackend": "rocksdb", 2023-11-28T14:32:06.664 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_log_collection_list_age": "60.000000", 2023-11-28T14:32:06.664 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2023-11-28T14:32:06.664 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_log_op_age": "5.000000", 2023-11-28T14:32:06.665 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_max_alloc_size": "0", 2023-11-28T14:32:06.665 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_max_blob_size": "0", 2023-11-28T14:32:06.665 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_max_blob_size_hdd": "65536", 2023-11-28T14:32:06.665 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_max_blob_size_ssd": "65536", 2023-11-28T14:32:06.665 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_max_defer_interval": "3.000000", 2023-11-28T14:32:06.665 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_max_deferred_txc": "32", 2023-11-28T14:32:06.665 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_min_alloc_size": "0", 2023-11-28T14:32:06.665 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_min_alloc_size_hdd": "4096", 2023-11-28T14:32:06.665 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_min_alloc_size_ssd": "4096", 2023-11-28T14:32:06.665 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_nid_prealloc": "1024", 2023-11-28T14:32:06.665 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_prefer_deferred_size": "0", 2023-11-28T14:32:06.665 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2023-11-28T14:32:06.665 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2023-11-28T14:32:06.666 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_qfsck_on_mount": "true", 2023-11-28T14:32:06.666 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_retry_disk_reads": "3", 2023-11-28T14:32:06.666 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_rocksdb_cf": "true", 2023-11-28T14:32:06.666 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2023-11-28T14:32:06.666 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2023-11-28T14:32:06.666 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_rocksdb_options_annex": "", 2023-11-28T14:32:06.666 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_spdk_coremask": "0x1", 2023-11-28T14:32:06.666 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_spdk_io_sleep": "5", 2023-11-28T14:32:06.666 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_spdk_max_io_completion": "0", 2023-11-28T14:32:06.666 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_spdk_mem": "512", 2023-11-28T14:32:06.666 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_sync_submit_transaction": "false", 2023-11-28T14:32:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_throttle_bytes": "67108864", 2023-11-28T14:32:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_throttle_cost_per_io": "0", 2023-11-28T14:32:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2023-11-28T14:32:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2023-11-28T14:32:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2023-11-28T14:32:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_throttle_trace_rate": "0.000000", 2023-11-28T14:32:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_tracing": "false", 2023-11-28T14:32:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2023-11-28T14:32:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2023-11-28T14:32:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_volume_selection_reserved": "0", 2023-11-28T14:32:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2023-11-28T14:32:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2023-11-28T14:32:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_warn_on_legacy_statfs": "true", 2023-11-28T14:32:06.668 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2023-11-28T14:32:06.668 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2023-11-28T14:32:06.668 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2023-11-28T14:32:06.668 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_zero_block_detection": "false", 2023-11-28T14:32:06.668 INFO:teuthology.orchestra.run.smithi055.stdout: "cephadm_path": "/usr/sbin/cephadm", 2023-11-28T14:32:06.668 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_action_update_interval": "2", 2023-11-28T14:32:06.668 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_directory_scan_interval": "10", 2023-11-28T14:32:06.668 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2023-11-28T14:32:06.668 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2023-11-28T14:32:06.668 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2023-11-28T14:32:06.668 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_mount_timeout": "10", 2023-11-28T14:32:06.668 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2023-11-28T14:32:06.668 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2023-11-28T14:32:06.669 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2023-11-28T14:32:06.669 INFO:teuthology.orchestra.run.smithi055.stdout: "cephsqlite_blocklist_dead_locker": "true", 2023-11-28T14:32:06.669 INFO:teuthology.orchestra.run.smithi055.stdout: "cephsqlite_lock_renewal_interval": "2000", 2023-11-28T14:32:06.669 INFO:teuthology.orchestra.run.smithi055.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2023-11-28T14:32:06.669 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_cluster_require_signatures": "false", 2023-11-28T14:32:06.669 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_cluster_require_version": "2", 2023-11-28T14:32:06.669 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_require_signatures": "false", 2023-11-28T14:32:06.669 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_require_version": "2", 2023-11-28T14:32:06.669 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_service_require_signatures": "false", 2023-11-28T14:32:06.669 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_service_require_version": "2", 2023-11-28T14:32:06.670 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_sign_messages": "true", 2023-11-28T14:32:06.670 INFO:teuthology.orchestra.run.smithi055.stdout: "chdir": "", 2023-11-28T14:32:06.670 INFO:teuthology.orchestra.run.smithi055.stdout: "client_acl_type": "", 2023-11-28T14:32:06.670 INFO:teuthology.orchestra.run.smithi055.stdout: "client_asio_thread_count": "2", 2023-11-28T14:32:06.670 INFO:teuthology.orchestra.run.smithi055.stdout: "client_cache_mid": "0.750000", 2023-11-28T14:32:06.670 INFO:teuthology.orchestra.run.smithi055.stdout: "client_cache_size": "16384", 2023-11-28T14:32:06.670 INFO:teuthology.orchestra.run.smithi055.stdout: "client_caps_release_delay": "5", 2023-11-28T14:32:06.670 INFO:teuthology.orchestra.run.smithi055.stdout: "client_check_pool_perm": "true", 2023-11-28T14:32:06.670 INFO:teuthology.orchestra.run.smithi055.stdout: "client_collect_and_send_global_metrics": "false", 2023-11-28T14:32:06.671 INFO:teuthology.orchestra.run.smithi055.stdout: "client_debug_force_sync_read": "false", 2023-11-28T14:32:06.671 INFO:teuthology.orchestra.run.smithi055.stdout: "client_debug_getattr_caps": "false", 2023-11-28T14:32:06.671 INFO:teuthology.orchestra.run.smithi055.stdout: "client_debug_inject_tick_delay": "0", 2023-11-28T14:32:06.671 INFO:teuthology.orchestra.run.smithi055.stdout: "client_die_on_failed_dentry_invalidate": "true", 2023-11-28T14:32:06.672 INFO:teuthology.orchestra.run.smithi055.stdout: "client_die_on_failed_remount": "false", 2023-11-28T14:32:06.672 INFO:teuthology.orchestra.run.smithi055.stdout: "client_dirsize_rbytes": "true", 2023-11-28T14:32:06.673 INFO:teuthology.orchestra.run.smithi055.stdout: "client_force_lazyio": "false", 2023-11-28T14:32:06.673 INFO:teuthology.orchestra.run.smithi055.stdout: "client_fs": "", 2023-11-28T14:32:06.673 INFO:teuthology.orchestra.run.smithi055.stdout: "client_inject_fixed_oldest_tid": "false", 2023-11-28T14:32:06.673 INFO:teuthology.orchestra.run.smithi055.stdout: "client_inject_release_failure": "false", 2023-11-28T14:32:06.673 INFO:teuthology.orchestra.run.smithi055.stdout: "client_max_inline_size": "4096", 2023-11-28T14:32:06.673 INFO:teuthology.orchestra.run.smithi055.stdout: "client_max_retries_on_remount_failure": "5", 2023-11-28T14:32:06.674 INFO:teuthology.orchestra.run.smithi055.stdout: "client_mds_namespace": "", 2023-11-28T14:32:06.674 INFO:teuthology.orchestra.run.smithi055.stdout: "client_metadata": "", 2023-11-28T14:32:06.674 INFO:teuthology.orchestra.run.smithi055.stdout: "client_mount_gid": "-1", 2023-11-28T14:32:06.674 INFO:teuthology.orchestra.run.smithi055.stdout: "client_mount_timeout": "300", 2023-11-28T14:32:06.674 INFO:teuthology.orchestra.run.smithi055.stdout: "client_mount_uid": "-1", 2023-11-28T14:32:06.674 INFO:teuthology.orchestra.run.smithi055.stdout: "client_mountpoint": "/", 2023-11-28T14:32:06.674 INFO:teuthology.orchestra.run.smithi055.stdout: "client_notify_timeout": "10", 2023-11-28T14:32:06.674 INFO:teuthology.orchestra.run.smithi055.stdout: "client_oc": "true", 2023-11-28T14:32:06.674 INFO:teuthology.orchestra.run.smithi055.stdout: "client_oc_max_dirty": "104857600", 2023-11-28T14:32:06.674 INFO:teuthology.orchestra.run.smithi055.stdout: "client_oc_max_dirty_age": "5.000000", 2023-11-28T14:32:06.675 INFO:teuthology.orchestra.run.smithi055.stdout: "client_oc_max_objects": "1000", 2023-11-28T14:32:06.675 INFO:teuthology.orchestra.run.smithi055.stdout: "client_oc_size": "209715200", 2023-11-28T14:32:06.675 INFO:teuthology.orchestra.run.smithi055.stdout: "client_oc_target_dirty": "8388608", 2023-11-28T14:32:06.675 INFO:teuthology.orchestra.run.smithi055.stdout: "client_permissions": "true", 2023-11-28T14:32:06.675 INFO:teuthology.orchestra.run.smithi055.stdout: "client_quota": "true", 2023-11-28T14:32:06.675 INFO:teuthology.orchestra.run.smithi055.stdout: "client_quota_df": "true", 2023-11-28T14:32:06.675 INFO:teuthology.orchestra.run.smithi055.stdout: "client_readahead_max_bytes": "0", 2023-11-28T14:32:06.675 INFO:teuthology.orchestra.run.smithi055.stdout: "client_readahead_max_periods": "4", 2023-11-28T14:32:06.675 INFO:teuthology.orchestra.run.smithi055.stdout: "client_readahead_min": "131072", 2023-11-28T14:32:06.675 INFO:teuthology.orchestra.run.smithi055.stdout: "client_reconnect_stale": "false", 2023-11-28T14:32:06.675 INFO:teuthology.orchestra.run.smithi055.stdout: "client_shutdown_timeout": "30", 2023-11-28T14:32:06.675 INFO:teuthology.orchestra.run.smithi055.stdout: "client_snapdir": ".snap", 2023-11-28T14:32:06.676 INFO:teuthology.orchestra.run.smithi055.stdout: "client_tick_interval": "1", 2023-11-28T14:32:06.676 INFO:teuthology.orchestra.run.smithi055.stdout: "client_trace": "", 2023-11-28T14:32:06.676 INFO:teuthology.orchestra.run.smithi055.stdout: "client_try_dentry_invalidate": "false", 2023-11-28T14:32:06.676 INFO:teuthology.orchestra.run.smithi055.stdout: "client_use_faked_inos": "false", 2023-11-28T14:32:06.676 INFO:teuthology.orchestra.run.smithi055.stdout: "client_use_random_mds": "false", 2023-11-28T14:32:06.676 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_graylog": "false", 2023-11-28T14:32:06.676 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_graylog_host": "127.0.0.1", 2023-11-28T14:32:06.676 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_graylog_port": "12201", 2023-11-28T14:32:06.676 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_monitors": "default=true", 2023-11-28T14:32:06.676 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_syslog": "false", 2023-11-28T14:32:06.676 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2023-11-28T14:32:06.676 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_syslog_level": "info", 2023-11-28T14:32:06.677 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "-", 2023-11-28T14:32:06.677 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_network": "", 2023-11-28T14:32:06.677 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_network_interface": "", 2023-11-28T14:32:06.677 INFO:teuthology.orchestra.run.smithi055.stdout: "compressor_zlib_isal": "false", 2023-11-28T14:32:06.677 INFO:teuthology.orchestra.run.smithi055.stdout: "compressor_zlib_level": "5", 2023-11-28T14:32:06.677 INFO:teuthology.orchestra.run.smithi055.stdout: "compressor_zlib_winsize": "-15", 2023-11-28T14:32:06.677 INFO:teuthology.orchestra.run.smithi055.stdout: "compressor_zstd_level": "1", 2023-11-28T14:32:06.677 INFO:teuthology.orchestra.run.smithi055.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2023-11-28T14:32:06.677 INFO:teuthology.orchestra.run.smithi055.stdout: "crash_dir": "/var/lib/ceph/crash", 2023-11-28T14:32:06.677 INFO:teuthology.orchestra.run.smithi055.stdout: "crimson_alien_op_num_threads": "6", 2023-11-28T14:32:06.677 INFO:teuthology.orchestra.run.smithi055.stdout: "crimson_alien_thread_cpu_cores": "", 2023-11-28T14:32:06.677 INFO:teuthology.orchestra.run.smithi055.stdout: "crimson_osd_obc_lru_size": "10", 2023-11-28T14:32:06.677 INFO:teuthology.orchestra.run.smithi055.stdout: "crimson_osd_scheduler_concurrency": "0", 2023-11-28T14:32:06.678 INFO:teuthology.orchestra.run.smithi055.stdout: "crimson_seastar_smp": "1", 2023-11-28T14:32:06.678 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_location": "", 2023-11-28T14:32:06.678 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_location_hook": "", 2023-11-28T14:32:06.678 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_location_hook_timeout": "10", 2023-11-28T14:32:06.678 INFO:teuthology.orchestra.run.smithi055.stdout: "daemonize": "false", 2023-11-28T14:32:06.678 INFO:teuthology.orchestra.run.smithi055.stdout: "daos_pool": "tank", 2023-11-28T14:32:06.678 INFO:teuthology.orchestra.run.smithi055.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2023-11-28T14:32:06.678 INFO:teuthology.orchestra.run.smithi055.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2023-11-28T14:32:06.678 INFO:teuthology.orchestra.run.smithi055.stdout: "dbstore_db_name_prefix": "dbstore", 2023-11-28T14:32:06.678 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_alienstore": "0/5", 2023-11-28T14:32:06.678 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_allow_any_pool_priority": "false", 2023-11-28T14:32:06.678 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_asok": "1/5", 2023-11-28T14:32:06.679 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_asok_assert_abort": "false", 2023-11-28T14:32:06.679 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_asserts_on_shutdown": "true", 2023-11-28T14:32:06.679 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_auth": "20/20", 2023-11-28T14:32:06.679 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_bdev": "1/3", 2023-11-28T14:32:06.679 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_bluefs": "1/5", 2023-11-28T14:32:06.679 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_bluestore": "1/5", 2023-11-28T14:32:06.679 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_buffer": "0/1", 2023-11-28T14:32:06.679 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_ceph_exporter": "1/5", 2023-11-28T14:32:06.679 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_cephfs_mirror": "0/5", 2023-11-28T14:32:06.679 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_cephsqlite": "0/5", 2023-11-28T14:32:06.679 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_client": "0/5", 2023-11-28T14:32:06.679 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_compressor": "1/5", 2023-11-28T14:32:06.680 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_context": "0/1", 2023-11-28T14:32:06.680 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_crimson_interrupt": "0/5", 2023-11-28T14:32:06.680 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_crush": "1/1", 2023-11-28T14:32:06.680 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_crypto": "1/5", 2023-11-28T14:32:06.680 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_cyanstore": "0/5", 2023-11-28T14:32:06.680 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_deliberately_leak_memory": "false", 2023-11-28T14:32:06.680 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_disable_randomized_ping": "false", 2023-11-28T14:32:06.680 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_dpdk": "1/5", 2023-11-28T14:32:06.680 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_eventtrace": "1/5", 2023-11-28T14:32:06.680 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_filer": "0/1", 2023-11-28T14:32:06.680 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_filestore": "1/3", 2023-11-28T14:32:06.680 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_finisher": "1/1", 2023-11-28T14:32:06.680 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_fuse": "1/5", 2023-11-28T14:32:06.681 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_heartbeat_testing_span": "0", 2023-11-28T14:32:06.681 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_heartbeatmap": "1/5", 2023-11-28T14:32:06.681 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_immutable_obj_cache": "0/5", 2023-11-28T14:32:06.681 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_javaclient": "1/5", 2023-11-28T14:32:06.681 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_journal": "1/3", 2023-11-28T14:32:06.681 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_journaler": "0/5", 2023-11-28T14:32:06.681 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_kstore": "1/5", 2023-11-28T14:32:06.681 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_lockdep": "0/1", 2023-11-28T14:32:06.681 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mclock": "1/5", 2023-11-28T14:32:06.681 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mds": "1/5", 2023-11-28T14:32:06.681 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mds_balancer": "1/5", 2023-11-28T14:32:06.681 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mds_locker": "1/5", 2023-11-28T14:32:06.682 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mds_log": "1/5", 2023-11-28T14:32:06.682 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mds_log_expire": "1/5", 2023-11-28T14:32:06.682 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mds_migrator": "1/5", 2023-11-28T14:32:06.682 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_memstore": "1/5", 2023-11-28T14:32:06.682 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mgr": "2/5", 2023-11-28T14:32:06.682 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mgrc": "1/5", 2023-11-28T14:32:06.682 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mon": "20/20", 2023-11-28T14:32:06.682 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_monc": "0/10", 2023-11-28T14:32:06.682 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_ms": "1/1", 2023-11-28T14:32:06.682 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_none": "0/5", 2023-11-28T14:32:06.682 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_objclass": "0/5", 2023-11-28T14:32:06.682 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_objectcacher": "0/5", 2023-11-28T14:32:06.682 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_objecter": "0/1", 2023-11-28T14:32:06.683 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_optracker": "0/5", 2023-11-28T14:32:06.683 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_osd": "1/5", 2023-11-28T14:32:06.683 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_paxos": "20/20", 2023-11-28T14:32:06.683 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_perfcounter": "1/5", 2023-11-28T14:32:06.683 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_prioritycache": "1/5", 2023-11-28T14:32:06.683 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rados": "0/5", 2023-11-28T14:32:06.683 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rbd": "0/5", 2023-11-28T14:32:06.683 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rbd_mirror": "0/5", 2023-11-28T14:32:06.683 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rbd_pwl": "0/5", 2023-11-28T14:32:06.683 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rbd_replay": "0/5", 2023-11-28T14:32:06.683 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_refs": "0/0", 2023-11-28T14:32:06.683 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_reserver": "1/1", 2023-11-28T14:32:06.683 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw": "1/5", 2023-11-28T14:32:06.684 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw_access": "1/5", 2023-11-28T14:32:06.684 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw_datacache": "1/5", 2023-11-28T14:32:06.684 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw_dbstore": "1/5", 2023-11-28T14:32:06.684 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw_flight": "1/5", 2023-11-28T14:32:06.684 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw_sync": "1/5", 2023-11-28T14:32:06.684 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rocksdb": "4/5", 2023-11-28T14:32:06.684 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore": "0/5", 2023-11-28T14:32:06.684 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_backref": "0/5", 2023-11-28T14:32:06.684 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_cache": "0/5", 2023-11-28T14:32:06.684 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_cleaner": "0/5", 2023-11-28T14:32:06.684 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_device": "0/5", 2023-11-28T14:32:06.684 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_epm": "0/5", 2023-11-28T14:32:06.685 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_fixedkv_tree": "0/5", 2023-11-28T14:32:06.685 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_journal": "0/5", 2023-11-28T14:32:06.685 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_lba": "0/5", 2023-11-28T14:32:06.685 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_odata": "0/5", 2023-11-28T14:32:06.685 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_omap": "0/5", 2023-11-28T14:32:06.685 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_onode": "0/5", 2023-11-28T14:32:06.685 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_t": "0/5", 2023-11-28T14:32:06.685 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_tm": "0/5", 2023-11-28T14:32:06.685 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_striper": "0/1", 2023-11-28T14:32:06.685 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_test": "0/5", 2023-11-28T14:32:06.685 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_throttle": "1/1", 2023-11-28T14:32:06.685 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_timer": "0/1", 2023-11-28T14:32:06.685 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_tp": "0/5", 2023-11-28T14:32:06.686 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_trace": "1/5", 2023-11-28T14:32:06.686 INFO:teuthology.orchestra.run.smithi055.stdout: "defer_client_eviction_on_laggy_osds": "true", 2023-11-28T14:32:06.686 INFO:teuthology.orchestra.run.smithi055.stdout: "device_failure_prediction_mode": "none", 2023-11-28T14:32:06.686 INFO:teuthology.orchestra.run.smithi055.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2023-11-28T14:32:06.686 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2023-11-28T14:32:06.686 INFO:teuthology.orchestra.run.smithi055.stdout: "err_to_graylog": "false", 2023-11-28T14:32:06.686 INFO:teuthology.orchestra.run.smithi055.stdout: "err_to_journald": "false", 2023-11-28T14:32:06.686 INFO:teuthology.orchestra.run.smithi055.stdout: "err_to_stderr": "true", 2023-11-28T14:32:06.686 INFO:teuthology.orchestra.run.smithi055.stdout: "err_to_syslog": "false", 2023-11-28T14:32:06.686 INFO:teuthology.orchestra.run.smithi055.stdout: "event_tracing": "false", 2023-11-28T14:32:06.686 INFO:teuthology.orchestra.run.smithi055.stdout: "exporter_addr": "0.0.0.0", 2023-11-28T14:32:06.687 INFO:teuthology.orchestra.run.smithi055.stdout: "exporter_http_port": "9926", 2023-11-28T14:32:06.687 INFO:teuthology.orchestra.run.smithi055.stdout: "exporter_prio_limit": "5", 2023-11-28T14:32:06.687 INFO:teuthology.orchestra.run.smithi055.stdout: "exporter_sock_dir": "/var/run/ceph/", 2023-11-28T14:32:06.687 INFO:teuthology.orchestra.run.smithi055.stdout: "exporter_sort_metrics": "true", 2023-11-28T14:32:06.687 INFO:teuthology.orchestra.run.smithi055.stdout: "exporter_stats_period": "5", 2023-11-28T14:32:06.687 INFO:teuthology.orchestra.run.smithi055.stdout: "fake_statfs_for_testing": "0", 2023-11-28T14:32:06.687 INFO:teuthology.orchestra.run.smithi055.stdout: "fatal_signal_handlers": "true", 2023-11-28T14:32:06.687 INFO:teuthology.orchestra.run.smithi055.stdout: "filer_max_purge_ops": "10", 2023-11-28T14:32:06.687 INFO:teuthology.orchestra.run.smithi055.stdout: "filer_max_truncate_ops": "128", 2023-11-28T14:32:06.687 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_apply_finisher_threads": "1", 2023-11-28T14:32:06.687 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_blackhole": "false", 2023-11-28T14:32:06.687 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_btrfs_clone_range": "true", 2023-11-28T14:32:06.687 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_btrfs_snap": "true", 2023-11-28T14:32:06.688 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_caller_concurrency": "10", 2023-11-28T14:32:06.688 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_collect_device_partition_information": "true", 2023-11-28T14:32:06.688 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_commit_timeout": "600.000000", 2023-11-28T14:32:06.688 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_debug_inject_read_err": "false", 2023-11-28T14:32:06.688 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_debug_omap_check": "false", 2023-11-28T14:32:06.688 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_debug_random_read_err": "0.000000", 2023-11-28T14:32:06.688 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_debug_verify_split": "false", 2023-11-28T14:32:06.688 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_dump_file": "", 2023-11-28T14:32:06.688 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2023-11-28T14:32:06.688 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_expected_throughput_ops": "200.000000", 2023-11-28T14:32:06.688 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fadvise": "true", 2023-11-28T14:32:06.688 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fail_eio": "true", 2023-11-28T14:32:06.688 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fd_cache_shards": "16", 2023-11-28T14:32:06.689 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fd_cache_size": "128", 2023-11-28T14:32:06.689 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fiemap": "false", 2023-11-28T14:32:06.689 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fiemap_threshold": "4096", 2023-11-28T14:32:06.689 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fsync_flushes_journal_data": "false", 2023-11-28T14:32:06.689 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_index_retry_probability": "0.000000", 2023-11-28T14:32:06.689 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_inject_stall": "0", 2023-11-28T14:32:06.689 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_journal_parallel": "false", 2023-11-28T14:32:06.689 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_journal_trailing": "false", 2023-11-28T14:32:06.689 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_journal_writeahead": "false", 2023-11-28T14:32:06.689 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_kill_at": "0", 2023-11-28T14:32:06.689 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_alloc_hint_size": "1048576", 2023-11-28T14:32:06.689 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattr_size": "0", 2023-11-28T14:32:06.690 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2023-11-28T14:32:06.690 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattr_size_other": "512", 2023-11-28T14:32:06.690 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2023-11-28T14:32:06.690 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattrs": "0", 2023-11-28T14:32:06.690 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2023-11-28T14:32:06.690 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattrs_other": "2", 2023-11-28T14:32:06.690 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattrs_xfs": "10", 2023-11-28T14:32:06.690 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_sync_interval": "5.000000", 2023-11-28T14:32:06.690 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_xattr_value_size": "0", 2023-11-28T14:32:06.690 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2023-11-28T14:32:06.690 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_xattr_value_size_other": "1024", 2023-11-28T14:32:06.690 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2023-11-28T14:32:06.691 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_merge_threshold": "-10", 2023-11-28T14:32:06.691 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_min_sync_interval": "0.010000", 2023-11-28T14:32:06.691 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_odsync_write": "false", 2023-11-28T14:32:06.691 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_omap_backend": "rocksdb", 2023-11-28T14:32:06.691 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_omap_backend_path": "", 2023-11-28T14:32:06.691 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_omap_header_cache_size": "1024", 2023-11-28T14:32:06.691 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_ondisk_finisher_threads": "1", 2023-11-28T14:32:06.691 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_op_thread_suicide_timeout": "180", 2023-11-28T14:32:06.691 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_op_thread_timeout": "60", 2023-11-28T14:32:06.691 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_op_threads": "2", 2023-11-28T14:32:06.691 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_punch_hole": "false", 2023-11-28T14:32:06.691 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2023-11-28T14:32:06.692 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2023-11-28T14:32:06.692 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2023-11-28T14:32:06.692 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_high_threshhold": "0.900000", 2023-11-28T14:32:06.692 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_low_threshhold": "0.300000", 2023-11-28T14:32:06.692 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_max_bytes": "104857600", 2023-11-28T14:32:06.692 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2023-11-28T14:32:06.692 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2023-11-28T14:32:06.692 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2023-11-28T14:32:06.692 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_max_ops": "50", 2023-11-28T14:32:06.692 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2023-11-28T14:32:06.692 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_seek_data_hole": "false", 2023-11-28T14:32:06.692 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_sloppy_crc": "false", 2023-11-28T14:32:06.692 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_sloppy_crc_block_size": "65536", 2023-11-28T14:32:06.693 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_splice": "false", 2023-11-28T14:32:06.693 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_split_multiple": "2", 2023-11-28T14:32:06.693 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_split_rand_factor": "20", 2023-11-28T14:32:06.693 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_update_to": "1000", 2023-11-28T14:32:06.693 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2023-11-28T14:32:06.693 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2023-11-28T14:32:06.693 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2023-11-28T14:32:06.693 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2023-11-28T14:32:06.693 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2023-11-28T14:32:06.693 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2023-11-28T14:32:06.693 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_enable": "true", 2023-11-28T14:32:06.693 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2023-11-28T14:32:06.693 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2023-11-28T14:32:06.694 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2023-11-28T14:32:06.694 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2023-11-28T14:32:06.694 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2023-11-28T14:32:06.694 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2023-11-28T14:32:06.694 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_xfs_extsize": "false", 2023-11-28T14:32:06.694 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_zfs_snap": "false", 2023-11-28T14:32:06.694 INFO:teuthology.orchestra.run.smithi055.stdout: "fio_dir": "/tmp/fio", 2023-11-28T14:32:06.694 INFO:teuthology.orchestra.run.smithi055.stdout: "fsid": "8639ce7e-f555-422d-b8f1-79137ef3334b", 2023-11-28T14:32:06.694 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_allow_other": "true", 2023-11-28T14:32:06.694 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_atomic_o_trunc": "true", 2023-11-28T14:32:06.694 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_big_writes": "true", 2023-11-28T14:32:06.694 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_debug": "false", 2023-11-28T14:32:06.695 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_default_permissions": "false", 2023-11-28T14:32:06.695 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_disable_pagecache": "false", 2023-11-28T14:32:06.695 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_max_write": "0", 2023-11-28T14:32:06.695 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_multithreaded": "true", 2023-11-28T14:32:06.695 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_require_active_mds": "true", 2023-11-28T14:32:06.695 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_set_user_groups": "true", 2023-11-28T14:32:06.695 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_splice_move": "true", 2023-11-28T14:32:06.695 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_splice_read": "true", 2023-11-28T14:32:06.695 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_splice_write": "true", 2023-11-28T14:32:06.695 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_syncfs_on_mksnap": "true", 2023-11-28T14:32:06.696 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_use_invalidate_cb": "true", 2023-11-28T14:32:06.696 INFO:teuthology.orchestra.run.smithi055.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2023-11-28T14:32:06.696 INFO:teuthology.orchestra.run.smithi055.stdout: "gss_target_name": "ceph", 2023-11-28T14:32:06.696 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_file": "", 2023-11-28T14:32:06.696 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_inject_failure": "0", 2023-11-28T14:32:06.696 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_interval": "5", 2023-11-28T14:32:06.696 INFO:teuthology.orchestra.run.smithi055.stdout: "host": "smithi055", 2023-11-28T14:32:06.696 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2023-11-28T14:32:06.696 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_max_inflight_ops": "128", 2023-11-28T14:32:06.696 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_max_size": "1073741824", 2023-11-28T14:32:06.696 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2023-11-28T14:32:06.696 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_bps_burst": "0", 2023-11-28T14:32:06.697 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2023-11-28T14:32:06.697 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_bps_limit": "0", 2023-11-28T14:32:06.697 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_iops_burst": "0", 2023-11-28T14:32:06.697 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2023-11-28T14:32:06.697 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_iops_limit": "0", 2023-11-28T14:32:06.697 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2023-11-28T14:32:06.697 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2023-11-28T14:32:06.697 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_watermark": "0.900000", 2023-11-28T14:32:06.697 INFO:teuthology.orchestra.run.smithi055.stdout: "inject_early_sigterm": "false", 2023-11-28T14:32:06.697 INFO:teuthology.orchestra.run.smithi055.stdout: "jaeger_agent_port": "6799", 2023-11-28T14:32:06.697 INFO:teuthology.orchestra.run.smithi055.stdout: "jaeger_tracing_enable": "false", 2023-11-28T14:32:06.699 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_aio": "true", 2023-11-28T14:32:06.700 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_align_min_size": "65536", 2023-11-28T14:32:06.700 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_block_align": "true", 2023-11-28T14:32:06.700 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_block_size": "4096", 2023-11-28T14:32:06.700 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_dio": "true", 2023-11-28T14:32:06.700 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_discard": "false", 2023-11-28T14:32:06.700 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_force_aio": "false", 2023-11-28T14:32:06.700 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_ignore_corruption": "false", 2023-11-28T14:32:06.700 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_max_write_bytes": "10485760", 2023-11-28T14:32:06.700 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_max_write_entries": "100", 2023-11-28T14:32:06.700 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_replay_from": "0", 2023-11-28T14:32:06.700 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_throttle_high_multiple": "0.000000", 2023-11-28T14:32:06.700 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_throttle_high_threshhold": "0.900000", 2023-11-28T14:32:06.701 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_throttle_low_threshhold": "0.600000", 2023-11-28T14:32:06.701 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_throttle_max_multiple": "0.000000", 2023-11-28T14:32:06.701 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_write_header_frequency": "0", 2023-11-28T14:32:06.701 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_zero_on_create": "false", 2023-11-28T14:32:06.701 INFO:teuthology.orchestra.run.smithi055.stdout: "journaler_prefetch_periods": "10", 2023-11-28T14:32:06.701 INFO:teuthology.orchestra.run.smithi055.stdout: "journaler_prezero_periods": "5", 2023-11-28T14:32:06.701 INFO:teuthology.orchestra.run.smithi055.stdout: "journaler_write_head_interval": "15", 2023-11-28T14:32:06.701 INFO:teuthology.orchestra.run.smithi055.stdout: "key": "", 2023-11-28T14:32:06.701 INFO:teuthology.orchestra.run.smithi055.stdout: "keyfile": "", 2023-11-28T14:32:06.701 INFO:teuthology.orchestra.run.smithi055.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2023-11-28T14:32:06.701 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_backend": "rocksdb", 2023-11-28T14:32:06.701 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_default_stripe_size": "65536", 2023-11-28T14:32:06.702 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_fsck_on_mount": "false", 2023-11-28T14:32:06.702 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_fsck_on_mount_deep": "true", 2023-11-28T14:32:06.702 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_max_bytes": "67108864", 2023-11-28T14:32:06.702 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_max_ops": "512", 2023-11-28T14:32:06.702 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_nid_prealloc": "1024", 2023-11-28T14:32:06.702 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_onode_map_size": "1024", 2023-11-28T14:32:06.702 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2023-11-28T14:32:06.702 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_sync_submit_transaction": "false", 2023-11-28T14:32:06.702 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_sync_transaction": "false", 2023-11-28T14:32:06.702 INFO:teuthology.orchestra.run.smithi055.stdout: "librados_thread_count": "2", 2023-11-28T14:32:06.702 INFO:teuthology.orchestra.run.smithi055.stdout: "lockdep": "false", 2023-11-28T14:32:06.702 INFO:teuthology.orchestra.run.smithi055.stdout: "lockdep_force_backtrace": "false", 2023-11-28T14:32:06.703 INFO:teuthology.orchestra.run.smithi055.stdout: "log_coarse_timestamps": "true", 2023-11-28T14:32:06.703 INFO:teuthology.orchestra.run.smithi055.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2023-11-28T14:32:06.703 INFO:teuthology.orchestra.run.smithi055.stdout: "log_flush_on_exit": "false", 2023-11-28T14:32:06.703 INFO:teuthology.orchestra.run.smithi055.stdout: "log_graylog_host": "127.0.0.1", 2023-11-28T14:32:06.703 INFO:teuthology.orchestra.run.smithi055.stdout: "log_graylog_port": "12201", 2023-11-28T14:32:06.703 INFO:teuthology.orchestra.run.smithi055.stdout: "log_max_new": "1000", 2023-11-28T14:32:06.703 INFO:teuthology.orchestra.run.smithi055.stdout: "log_max_recent": "10000", 2023-11-28T14:32:06.703 INFO:teuthology.orchestra.run.smithi055.stdout: "log_stderr_prefix": "", 2023-11-28T14:32:06.703 INFO:teuthology.orchestra.run.smithi055.stdout: "log_stop_at_utilization": "0.970000", 2023-11-28T14:32:06.703 INFO:teuthology.orchestra.run.smithi055.stdout: "log_to_file": "true", 2023-11-28T14:32:06.703 INFO:teuthology.orchestra.run.smithi055.stdout: "log_to_graylog": "false", 2023-11-28T14:32:06.703 INFO:teuthology.orchestra.run.smithi055.stdout: "log_to_journald": "false", 2023-11-28T14:32:06.703 INFO:teuthology.orchestra.run.smithi055.stdout: "log_to_stderr": "false", 2023-11-28T14:32:06.704 INFO:teuthology.orchestra.run.smithi055.stdout: "log_to_syslog": "false", 2023-11-28T14:32:06.704 INFO:teuthology.orchestra.run.smithi055.stdout: "mandatory_topic_permissions": "false", 2023-11-28T14:32:06.704 INFO:teuthology.orchestra.run.smithi055.stdout: "max_rotating_auth_attempts": "10", 2023-11-28T14:32:06.704 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2023-11-28T14:32:06.704 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_action_on_write_error": "1", 2023-11-28T14:32:06.704 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_alternate_name_max": "8192", 2023-11-28T14:32:06.704 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_asio_thread_count": "2", 2023-11-28T14:32:06.704 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_export_pin": "true", 2023-11-28T14:32:06.704 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_fragment_dirs": "true", 2023-11-28T14:32:06.704 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2023-11-28T14:32:06.704 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_fragment_interval": "5", 2023-11-28T14:32:06.705 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_fragment_size_max": "100000", 2023-11-28T14:32:06.705 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_idle_threshold": "0.000000", 2023-11-28T14:32:06.705 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_interval": "10", 2023-11-28T14:32:06.705 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_max": "-1", 2023-11-28T14:32:06.705 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_max_until": "-1", 2023-11-28T14:32:06.705 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_merge_size": "50", 2023-11-28T14:32:06.705 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_midchunk": "0.300000", 2023-11-28T14:32:06.705 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_min_rebalance": "0.100000", 2023-11-28T14:32:06.705 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_min_start": "0.200000", 2023-11-28T14:32:06.705 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_minchunk": "0.001000", 2023-11-28T14:32:06.705 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_mode": "0", 2023-11-28T14:32:06.705 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_need_max": "1.200000", 2023-11-28T14:32:06.706 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_need_min": "0.800000", 2023-11-28T14:32:06.706 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_overload_epochs": "2", 2023-11-28T14:32:06.706 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_replicate_threshold": "8000.000000", 2023-11-28T14:32:06.706 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_sample_interval": "3.000000", 2023-11-28T14:32:06.706 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_split_bits": "3", 2023-11-28T14:32:06.706 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_split_rd": "25000.000000", 2023-11-28T14:32:06.706 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_split_size": "10000", 2023-11-28T14:32:06.706 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_split_wr": "10000.000000", 2023-11-28T14:32:06.706 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_target_decay": "10.000000", 2023-11-28T14:32:06.706 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2023-11-28T14:32:06.706 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_beacon_grace": "15.000000", 2023-11-28T14:32:06.706 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_beacon_interval": "4.000000", 2023-11-28T14:32:06.707 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_beacon_mon_down_grace": "60", 2023-11-28T14:32:06.707 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_memory_limit": "4294967296", 2023-11-28T14:32:06.707 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_mid": "0.700000", 2023-11-28T14:32:06.707 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_release_free_interval": "10", 2023-11-28T14:32:06.707 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_reservation": "0.050000", 2023-11-28T14:32:06.707 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_trim_decay_rate": "1.000000", 2023-11-28T14:32:06.707 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_trim_interval": "1", 2023-11-28T14:32:06.707 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_trim_threshold": "262144", 2023-11-28T14:32:06.707 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2023-11-28T14:32:06.707 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2023-11-28T14:32:06.707 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_client_delegate_inos_pct": "50", 2023-11-28T14:32:06.707 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_client_prealloc_inos": "1000", 2023-11-28T14:32:06.707 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2023-11-28T14:32:06.708 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_connect_bootstrapping": "false", 2023-11-28T14:32:06.708 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_damage_table_max_entries": "10000", 2023-11-28T14:32:06.708 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2023-11-28T14:32:06.708 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_debug_auth_pins": "false", 2023-11-28T14:32:06.708 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_debug_frag": "false", 2023-11-28T14:32:06.708 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_debug_scatterstat": "false", 2023-11-28T14:32:06.708 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_debug_subtrees": "false", 2023-11-28T14:32:06.708 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_decay_halflife": "5.000000", 2023-11-28T14:32:06.708 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_default_dir_hash": "2", 2023-11-28T14:32:06.708 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_defer_session_stale": "true", 2023-11-28T14:32:06.708 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_deny_all_reconnect": "false", 2023-11-28T14:32:06.708 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dir_keys_per_op": "16384", 2023-11-28T14:32:06.709 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dir_max_commit_size": "10", 2023-11-28T14:32:06.709 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dir_max_entries": "0", 2023-11-28T14:32:06.709 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dir_prefetch": "true", 2023-11-28T14:32:06.709 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dirstat_min_interval": "1.000000", 2023-11-28T14:32:06.709 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dump_cache_after_rejoin": "false", 2023-11-28T14:32:06.709 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dump_cache_on_map": "false", 2023-11-28T14:32:06.709 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dump_cache_threshold_file": "0", 2023-11-28T14:32:06.709 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2023-11-28T14:32:06.709 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_early_reply": "true", 2023-11-28T14:32:06.709 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_enable_op_tracker": "true", 2023-11-28T14:32:06.709 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_enforce_unique_name": "true", 2023-11-28T14:32:06.709 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_export_ephemeral_distributed": "true", 2023-11-28T14:32:06.710 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2023-11-28T14:32:06.710 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_export_ephemeral_random": "true", 2023-11-28T14:32:06.710 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_export_ephemeral_random_max": "0.010000", 2023-11-28T14:32:06.710 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_extraordinary_events_dump_interval": "0", 2023-11-28T14:32:06.710 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_forward_all_requests_to_auth": "false", 2023-11-28T14:32:06.710 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_freeze_tree_timeout": "30.000000", 2023-11-28T14:32:06.710 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_fscrypt_last_block_max_size": "4096", 2023-11-28T14:32:06.710 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_go_bad_corrupt_dentry": "true", 2023-11-28T14:32:06.710 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2023-11-28T14:32:06.710 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_health_cache_threshold": "1.500000", 2023-11-28T14:32:06.710 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_health_summarize_threshold": "10", 2023-11-28T14:32:06.710 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_heartbeat_grace": "15.000000", 2023-11-28T14:32:06.710 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_heartbeat_reset_grace": "1000", 2023-11-28T14:32:06.711 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_inject_health_dummy": "false", 2023-11-28T14:32:06.711 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2023-11-28T14:32:06.711 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_inject_migrator_session_race": "false", 2023-11-28T14:32:06.711 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2023-11-28T14:32:06.711 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_inject_skip_replaying_inotable": "false", 2023-11-28T14:32:06.711 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2023-11-28T14:32:06.711 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_join_fs": "", 2023-11-28T14:32:06.711 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_journal_format": "1", 2023-11-28T14:32:06.711 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_create_at": "0", 2023-11-28T14:32:06.711 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_export_at": "0", 2023-11-28T14:32:06.711 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_import_at": "0", 2023-11-28T14:32:06.711 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_journal_at": "0", 2023-11-28T14:32:06.712 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_journal_expire_at": "0", 2023-11-28T14:32:06.712 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_journal_replay_at": "0", 2023-11-28T14:32:06.712 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_link_at": "0", 2023-11-28T14:32:06.712 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_mdstable_at": "0", 2023-11-28T14:32:06.712 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_openc_at": "0", 2023-11-28T14:32:06.712 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_rename_at": "0", 2023-11-28T14:32:06.712 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_shutdown_at": "0", 2023-11-28T14:32:06.712 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_skip_replaying_inotable": "false", 2023-11-28T14:32:06.712 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_event_large_threshold": "524288", 2023-11-28T14:32:06.712 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_events_per_segment": "1024", 2023-11-28T14:32:06.712 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_major_segment_event_ratio": "12", 2023-11-28T14:32:06.712 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_max_events": "-1", 2023-11-28T14:32:06.713 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_max_segments": "128", 2023-11-28T14:32:06.713 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_pause": "false", 2023-11-28T14:32:06.713 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_segment_size": "0", 2023-11-28T14:32:06.713 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_skip_corrupt_events": "false", 2023-11-28T14:32:06.713 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_skip_unbounded_events": "false", 2023-11-28T14:32:06.713 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_warn_factor": "2.000000", 2023-11-28T14:32:06.713 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_caps_per_client": "1048576", 2023-11-28T14:32:06.713 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_completed_flushes": "100000", 2023-11-28T14:32:06.713 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_completed_requests": "100000", 2023-11-28T14:32:06.713 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_export_size": "20971520", 2023-11-28T14:32:06.713 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_file_recover": "32", 2023-11-28T14:32:06.713 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_purge_files": "64", 2023-11-28T14:32:06.714 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_purge_ops": "8192", 2023-11-28T14:32:06.714 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2023-11-28T14:32:06.714 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_scrub_ops_in_progress": "5", 2023-11-28T14:32:06.714 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_snaps_per_dir": "100", 2023-11-28T14:32:06.714 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_metrics_update_interval": "2", 2023-11-28T14:32:06.714 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_min_caps_per_client": "100", 2023-11-28T14:32:06.714 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_min_caps_working_set": "10000", 2023-11-28T14:32:06.714 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_mon_shutdown_timeout": "5.000000", 2023-11-28T14:32:06.714 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_numa_node": "-1", 2023-11-28T14:32:06.714 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_oft_prefetch_dirfrags": "false", 2023-11-28T14:32:06.714 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_op_complaint_time": "30.000000", 2023-11-28T14:32:06.714 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_op_history_duration": "600", 2023-11-28T14:32:06.715 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_op_history_size": "20", 2023-11-28T14:32:06.715 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_op_history_slow_op_size": "20", 2023-11-28T14:32:06.715 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_op_history_slow_op_threshold": "10", 2023-11-28T14:32:06.715 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_op_log_threshold": "5", 2023-11-28T14:32:06.715 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_ping_grace": "15", 2023-11-28T14:32:06.715 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_ping_interval": "5", 2023-11-28T14:32:06.715 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2023-11-28T14:32:06.715 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_recall_global_max_decay_threshold": "131072", 2023-11-28T14:32:06.715 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_recall_max_caps": "30000", 2023-11-28T14:32:06.715 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_recall_max_decay_rate": "1.500000", 2023-11-28T14:32:06.715 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_recall_max_decay_threshold": "131072", 2023-11-28T14:32:06.715 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_recall_warning_decay_rate": "60.000000", 2023-11-28T14:32:06.715 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_recall_warning_threshold": "262144", 2023-11-28T14:32:06.716 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_reconnect_timeout": "45.000000", 2023-11-28T14:32:06.716 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_replay_interval": "1.000000", 2023-11-28T14:32:06.716 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_replay_unsafe_with_closed_session": "false", 2023-11-28T14:32:06.716 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_request_load_average_decay_rate": "60.000000", 2023-11-28T14:32:06.716 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_root_ino_gid": "0", 2023-11-28T14:32:06.716 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_root_ino_uid": "0", 2023-11-28T14:32:06.716 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_scatter_nudge_interval": "5.000000", 2023-11-28T14:32:06.716 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_blocklist_on_evict": "true", 2023-11-28T14:32:06.716 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_blocklist_on_timeout": "true", 2023-11-28T14:32:06.716 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2023-11-28T14:32:06.716 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_cache_liveness_magnitude": "10", 2023-11-28T14:32:06.716 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2023-11-28T14:32:06.717 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_cap_acquisition_throttle": "100000", 2023-11-28T14:32:06.717 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2023-11-28T14:32:06.717 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_metadata_threshold": "16777216", 2023-11-28T14:32:06.717 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_sessionmap_keys_per_op": "1024", 2023-11-28T14:32:06.717 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_shutdown_check": "0", 2023-11-28T14:32:06.717 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_skip_ino": "0", 2023-11-28T14:32:06.717 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_sleep_rank_change": "0.000000", 2023-11-28T14:32:06.717 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_snap_max_uid": "4294967294", 2023-11-28T14:32:06.717 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_snap_min_uid": "0", 2023-11-28T14:32:06.717 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_snap_rstat": "false", 2023-11-28T14:32:06.717 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_standby_replay_damaged": "false", 2023-11-28T14:32:06.717 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_symlink_recovery": "true", 2023-11-28T14:32:06.718 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_task_status_update_interval": "2.000000", 2023-11-28T14:32:06.718 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_thrash_exports": "0", 2023-11-28T14:32:06.718 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_thrash_fragments": "0", 2023-11-28T14:32:06.718 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_tick_interval": "5.000000", 2023-11-28T14:32:06.718 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_valgrind_exit": "false", 2023-11-28T14:32:06.718 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_verify_backtrace": "1", 2023-11-28T14:32:06.718 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_verify_scatter": "false", 2023-11-28T14:32:06.718 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_wipe_ino_prealloc": "false", 2023-11-28T14:32:06.718 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_wipe_sessions": "false", 2023-11-28T14:32:06.718 INFO:teuthology.orchestra.run.smithi055.stdout: "mempool_debug": "false", 2023-11-28T14:32:06.719 INFO:teuthology.orchestra.run.smithi055.stdout: "memstore_debug_omit_block_device_write": "false", 2023-11-28T14:32:06.719 INFO:teuthology.orchestra.run.smithi055.stdout: "memstore_device_bytes": "1073741824", 2023-11-28T14:32:06.719 INFO:teuthology.orchestra.run.smithi055.stdout: "memstore_page_set": "false", 2023-11-28T14:32:06.719 INFO:teuthology.orchestra.run.smithi055.stdout: "memstore_page_size": "65536", 2023-11-28T14:32:06.719 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_client_bytes": "134217728", 2023-11-28T14:32:06.719 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_client_messages": "512", 2023-11-28T14:32:06.719 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2023-11-28T14:32:06.719 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_connect_retry_interval": "1.000000", 2023-11-28T14:32:06.719 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2023-11-28T14:32:06.719 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2023-11-28T14:32:06.719 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_disabled_modules": "", 2023-11-28T14:32:06.719 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_initial_modules": "restful iostat nfs", 2023-11-28T14:32:06.720 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_max_pg_num_change": "128", 2023-11-28T14:32:06.720 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_mds_bytes": "134217728", 2023-11-28T14:32:06.720 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_mds_messages": "128", 2023-11-28T14:32:06.720 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2023-11-28T14:32:06.720 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_mon_bytes": "134217728", 2023-11-28T14:32:06.720 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_mon_messages": "128", 2023-11-28T14:32:06.720 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_osd_bytes": "536870912", 2023-11-28T14:32:06.720 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_osd_messages": "8192", 2023-11-28T14:32:06.720 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_pool": "true", 2023-11-28T14:32:06.720 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_service_beacon_grace": "60.000000", 2023-11-28T14:32:06.720 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_standby_modules": "true", 2023-11-28T14:32:06.721 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_stats_period": "5", 2023-11-28T14:32:06.721 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_stats_threshold": "5", 2023-11-28T14:32:06.721 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_tick_period": "2", 2023-11-28T14:32:06.721 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_ttl_cache_expire_seconds": "0", 2023-11-28T14:32:06.721 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_accept_timeout_factor": "2.000000", 2023-11-28T14:32:06.721 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_allow_pool_delete": "true", 2023-11-28T14:32:06.721 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_allow_pool_size_one": "true", 2023-11-28T14:32:06.721 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_auth_validate_all_caps": "true", 2023-11-28T14:32:06.721 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2023-11-28T14:32:06.721 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2023-11-28T14:32:06.721 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_bytes": "104857600", 2023-11-28T14:32:06.721 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_directed_command_retry": "2", 2023-11-28T14:32:06.722 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_hunt_interval": "3.000000", 2023-11-28T14:32:06.722 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2023-11-28T14:32:06.722 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2023-11-28T14:32:06.722 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2023-11-28T14:32:06.722 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_hunt_parallel": "3", 2023-11-28T14:32:06.722 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_log_interval": "1.000000", 2023-11-28T14:32:06.722 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_max_log_entries_per_message": "1000", 2023-11-28T14:32:06.722 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_ping_interval": "10.000000", 2023-11-28T14:32:06.722 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_ping_timeout": "30.000000", 2023-11-28T14:32:06.722 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_target_rank": "-1", 2023-11-28T14:32:06.722 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_clock_drift_allowed": "1.000000", 2023-11-28T14:32:06.722 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2023-11-28T14:32:06.723 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2023-11-28T14:32:06.723 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_file_level": "debug", 2023-11-28T14:32:06.723 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_file": "true", 2023-11-28T14:32:06.723 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_graylog": "false", 2023-11-28T14:32:06.723 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2023-11-28T14:32:06.723 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_graylog_port": "12201", 2023-11-28T14:32:06.723 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_journald": "false", 2023-11-28T14:32:06.723 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_stderr": "false", 2023-11-28T14:32:06.723 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_syslog": "default=false", 2023-11-28T14:32:06.723 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2023-11-28T14:32:06.723 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_syslog_level": "info", 2023-11-28T14:32:06.723 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_compact_on_bootstrap": "false", 2023-11-28T14:32:06.724 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_compact_on_start": "false", 2023-11-28T14:32:06.724 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_compact_on_trim": "true", 2023-11-28T14:32:06.724 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_con_tracker_persist_interval": "10", 2023-11-28T14:32:06.724 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_con_tracker_score_halflife": "43200", 2023-11-28T14:32:06.724 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_config_key_max_entry_size": "65536", 2023-11-28T14:32:06.724 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cpu_threads": "4", 2023-11-28T14:32:06.724 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_crush_min_required_version": "hammer", 2023-11-28T14:32:06.724 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_daemon_bytes": "419430400", 2023-11-28T14:32:06.724 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2023-11-28T14:32:06.724 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_data_avail_crit": "5", 2023-11-28T14:32:06.724 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_data_avail_warn": "5", 2023-11-28T14:32:06.724 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_data_size_warn": "16106127360", 2023-11-28T14:32:06.725 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_block_osdmap_trim": "false", 2023-11-28T14:32:06.725 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_deprecated_as_obsolete": "false", 2023-11-28T14:32:06.725 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_dump_json": "false", 2023-11-28T14:32:06.725 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2023-11-28T14:32:06.725 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_dump_transactions": "false", 2023-11-28T14:32:06.725 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_extra_checks": "false", 2023-11-28T14:32:06.725 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_no_initial_persistent_features": "false", 2023-11-28T14:32:06.725 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2023-11-28T14:32:06.725 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_no_require_quincy": "false", 2023-11-28T14:32:06.725 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_no_require_reef": "false", 2023-11-28T14:32:06.725 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2023-11-28T14:32:06.725 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_delta_reset_interval": "10.000000", 2023-11-28T14:32:06.726 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_dns_srv_name": "ceph-mon", 2023-11-28T14:32:06.726 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_down_mkfs_grace": "120", 2023-11-28T14:32:06.726 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_election_default_strategy": "1", 2023-11-28T14:32:06.726 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_election_timeout": "5.000000", 2023-11-28T14:32:06.726 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2023-11-28T14:32:06.726 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_elector_ping_divisor": "2", 2023-11-28T14:32:06.726 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_elector_ping_timeout": "2.000000", 2023-11-28T14:32:06.726 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_enable_op_tracker": "true", 2023-11-28T14:32:06.726 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_fake_pool_delete": "false", 2023-11-28T14:32:06.726 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_force_quorum_join": "false", 2023-11-28T14:32:06.726 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_globalid_prealloc": "10000", 2023-11-28T14:32:06.726 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_health_detail_to_clog": "false", 2023-11-28T14:32:06.727 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_health_log_update_period": "5", 2023-11-28T14:32:06.727 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_health_max_detail": "50", 2023-11-28T14:32:06.727 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_health_to_clog": "true", 2023-11-28T14:32:06.727 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_health_to_clog_interval": "600", 2023-11-28T14:32:06.727 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2023-11-28T14:32:06.727 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_host": "172.21.15.55", 2023-11-28T14:32:06.727 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_host_override": "", 2023-11-28T14:32:06.727 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_initial_members": "", 2023-11-28T14:32:06.727 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2023-11-28T14:32:06.727 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2023-11-28T14:32:06.727 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_inject_transaction_delay_max": "10.000000", 2023-11-28T14:32:06.728 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2023-11-28T14:32:06.728 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_keyvaluedb": "rocksdb", 2023-11-28T14:32:06.728 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_lease": "5.000000", 2023-11-28T14:32:06.728 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2023-11-28T14:32:06.728 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_lease_renew_interval_factor": "0.600000", 2023-11-28T14:32:06.728 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_log_full_interval": "50", 2023-11-28T14:32:06.728 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_log_max": "10000", 2023-11-28T14:32:06.728 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_log_max_summary": "50", 2023-11-28T14:32:06.728 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_log_entries_per_event": "4096", 2023-11-28T14:32:06.728 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_log_epochs": "500", 2023-11-28T14:32:06.728 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_mdsmap_epochs": "500", 2023-11-28T14:32:06.728 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_mgrmap_epochs": "500", 2023-11-28T14:32:06.729 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_osd": "10000", 2023-11-28T14:32:06.729 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_pg_per_osd": "10000", 2023-11-28T14:32:06.729 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_pool_pg_num": "65536", 2023-11-28T14:32:06.729 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_snap_prune_per_epoch": "100", 2023-11-28T14:32:06.729 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mds_blocklist_interval": "86400.000000", 2023-11-28T14:32:06.729 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mds_force_trim_to": "0", 2023-11-28T14:32:06.729 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mds_skip_sanity": "false", 2023-11-28T14:32:06.729 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_memory_autotune": "true", 2023-11-28T14:32:06.729 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_memory_target": "2147483648", 2023-11-28T14:32:06.729 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mgr_beacon_grace": "30", 2023-11-28T14:32:06.729 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2023-11-28T14:32:06.729 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mgr_digest_period": "5", 2023-11-28T14:32:06.730 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mgr_inactive_grace": "60", 2023-11-28T14:32:06.730 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mgr_mkfs_grace": "240", 2023-11-28T14:32:06.730 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2023-11-28T14:32:06.730 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_min_osdmap_epochs": "50", 2023-11-28T14:32:06.730 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_op_complaint_time": "30", 2023-11-28T14:32:06.730 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_op_history_duration": "600", 2023-11-28T14:32:06.730 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_op_history_size": "20", 2023-11-28T14:32:06.730 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_op_history_slow_op_size": "20", 2023-11-28T14:32:06.730 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_op_history_slow_op_threshold": "10", 2023-11-28T14:32:06.730 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_op_log_threshold": "5", 2023-11-28T14:32:06.730 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_adjust_down_out_interval": "true", 2023-11-28T14:32:06.730 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2023-11-28T14:32:06.731 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2023-11-28T14:32:06.731 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_auto_mark_in": "false", 2023-11-28T14:32:06.731 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_auto_mark_new_in": "true", 2023-11-28T14:32:06.731 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2023-11-28T14:32:06.731 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2023-11-28T14:32:06.731 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_cache_size": "500", 2023-11-28T14:32:06.731 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_cache_size_min": "134217728", 2023-11-28T14:32:06.731 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_crush_smoke_test": "true", 2023-11-28T14:32:06.731 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_destroyed_out_interval": "600", 2023-11-28T14:32:06.731 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_down_out_interval": "600", 2023-11-28T14:32:06.731 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_down_out_subtree_limit": "rack", 2023-11-28T14:32:06.731 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2023-11-28T14:32:06.732 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_force_trim_to": "0", 2023-11-28T14:32:06.732 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_full_ratio": "0.950000", 2023-11-28T14:32:06.732 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2023-11-28T14:32:06.732 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_laggy_halflife": "3600", 2023-11-28T14:32:06.732 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_laggy_max_interval": "300", 2023-11-28T14:32:06.732 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_laggy_weight": "0.300000", 2023-11-28T14:32:06.732 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2023-11-28T14:32:06.732 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_max_creating_pgs": "1024", 2023-11-28T14:32:06.732 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_max_initial_pgs": "1024", 2023-11-28T14:32:06.732 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_min_down_reporters": "2", 2023-11-28T14:32:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_min_in_ratio": "0.750000", 2023-11-28T14:32:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_min_up_ratio": "0.300000", 2023-11-28T14:32:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_nearfull_ratio": "0.850000", 2023-11-28T14:32:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_prime_pg_temp": "true", 2023-11-28T14:32:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2023-11-28T14:32:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2023-11-28T14:32:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_report_timeout": "900", 2023-11-28T14:32:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_reporter_subtree_level": "osd", 2023-11-28T14:32:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2023-11-28T14:32:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_warn_num_repaired": "10", 2023-11-28T14:32:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_warn_op_age": "32.000000", 2023-11-28T14:32:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osdmap_full_prune_enabled": "true", 2023-11-28T14:32:06.734 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osdmap_full_prune_interval": "2", 2023-11-28T14:32:06.734 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osdmap_full_prune_min": "15", 2023-11-28T14:32:06.734 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osdmap_full_prune_txsize": "2", 2023-11-28T14:32:06.734 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2023-11-28T14:32:06.734 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pg_stuck_threshold": "60", 2023-11-28T14:32:06.734 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2023-11-28T14:32:06.734 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pg_warn_min_objects": "10000", 2023-11-28T14:32:06.734 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pg_warn_min_per_osd": "0", 2023-11-28T14:32:06.734 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pg_warn_min_pool_objects": "1000", 2023-11-28T14:32:06.734 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pool_quota_crit_threshold": "0", 2023-11-28T14:32:06.734 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pool_quota_warn_threshold": "0", 2023-11-28T14:32:06.735 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_probe_timeout": "2.000000", 2023-11-28T14:32:06.735 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_reweight_max_change": "0.050000", 2023-11-28T14:32:06.735 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_reweight_max_osds": "4", 2023-11-28T14:32:06.735 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_reweight_min_bytes_per_osd": "10", 2023-11-28T14:32:06.735 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_reweight_min_pgs_per_osd": "4", 2023-11-28T14:32:06.735 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2023-11-28T14:32:06.735 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2023-11-28T14:32:06.735 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2023-11-28T14:32:06.735 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_scrub_interval": "86400", 2023-11-28T14:32:06.735 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_scrub_max_keys": "100", 2023-11-28T14:32:06.735 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_scrub_timeout": "300", 2023-11-28T14:32:06.736 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_session_timeout": "300", 2023-11-28T14:32:06.736 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_smart_report_timeout": "5", 2023-11-28T14:32:06.736 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_stat_smooth_intervals": "6", 2023-11-28T14:32:06.736 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2023-11-28T14:32:06.736 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_stretch_pool_min_size": "2", 2023-11-28T14:32:06.736 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_stretch_pool_size": "4", 2023-11-28T14:32:06.736 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2023-11-28T14:32:06.736 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_subscribe_interval": "86400.000000", 2023-11-28T14:32:06.736 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_sync_debug": "false", 2023-11-28T14:32:06.736 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_sync_max_payload_keys": "2000", 2023-11-28T14:32:06.736 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_sync_max_payload_size": "1048576", 2023-11-28T14:32:06.736 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_sync_provider_kill_at": "0", 2023-11-28T14:32:06.736 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_sync_requester_kill_at": "0", 2023-11-28T14:32:06.737 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_sync_timeout": "60.000000", 2023-11-28T14:32:06.737 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_target_pg_per_osd": "100", 2023-11-28T14:32:06.737 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_tick_interval": "5", 2023-11-28T14:32:06.737 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_timecheck_interval": "300.000000", 2023-11-28T14:32:06.737 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_timecheck_skew_interval": "30.000000", 2023-11-28T14:32:06.737 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_use_min_delay_socket": "false", 2023-11-28T14:32:06.737 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_older_version_delay": "604800", 2023-11-28T14:32:06.737 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2023-11-28T14:32:06.737 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2023-11-28T14:32:06.737 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2023-11-28T14:32:06.737 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_filestore_osds": "false", 2023-11-28T14:32:06.738 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2023-11-28T14:32:06.738 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2023-11-28T14:32:06.738 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2023-11-28T14:32:06.738 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_misplaced": "false", 2023-11-28T14:32:06.738 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2023-11-28T14:32:06.738 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_older_version": "true", 2023-11-28T14:32:06.738 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2023-11-28T14:32:06.738 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_pool_no_app": "true", 2023-11-28T14:32:06.738 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_pool_no_redundancy": "false", 2023-11-28T14:32:06.738 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2023-11-28T14:32:06.738 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2023-11-28T14:32:06.739 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2023-11-28T14:32:06.739 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_too_few_osds": "false", 2023-11-28T14:32:06.739 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2023-11-28T14:32:06.739 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2023-11-28T14:32:06.739 INFO:teuthology.orchestra.run.smithi055.stdout: "monmap": "", 2023-11-28T14:32:06.739 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-11-28T14:32:06.739 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2023-11-28T14:32:06.739 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2023-11-28T14:32:06.739 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-11-28T14:32:06.739 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2023-11-28T14:32:06.739 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2023-11-28T14:32:06.740 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_tracing_enabled": "false", 2023-11-28T14:32:06.740 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_op_threads": "3", 2023-11-28T14:32:06.740 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_buffer_size": "131072", 2023-11-28T14:32:06.740 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_cm": "false", 2023-11-28T14:32:06.740 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_device_name": "", 2023-11-28T14:32:06.740 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_dscp": "96", 2023-11-28T14:32:06.740 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_enable_hugepage": "false", 2023-11-28T14:32:06.740 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_gid_idx": "0", 2023-11-28T14:32:06.740 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_local_gid": "", 2023-11-28T14:32:06.740 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_polling_us": "1000", 2023-11-28T14:32:06.740 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_port_num": "1", 2023-11-28T14:32:06.740 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_receive_buffers": "32768", 2023-11-28T14:32:06.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_receive_queue_len": "4096", 2023-11-28T14:32:06.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_roce_ver": "1", 2023-11-28T14:32:06.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_send_buffers": "1024", 2023-11-28T14:32:06.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_sl": "3", 2023-11-28T14:32:06.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_support_srq": "true", 2023-11-28T14:32:06.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_type": "ib", 2023-11-28T14:32:06.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_reap_threshold": "5", 2023-11-28T14:32:06.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_before_connect": "false", 2023-11-28T14:32:06.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_ipv4": "true", 2023-11-28T14:32:06.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_ipv6": "false", 2023-11-28T14:32:06.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_msgr1": "true", 2023-11-28T14:32:06.742 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_msgr2": "true", 2023-11-28T14:32:06.742 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_port_max": "7568", 2023-11-28T14:32:06.742 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_port_min": "6800", 2023-11-28T14:32:06.742 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_prefer_ipv4": "false", 2023-11-28T14:32:06.742 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_retry_count": "3", 2023-11-28T14:32:06.742 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_retry_delay": "5", 2023-11-28T14:32:06.742 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_blackhole_client": "false", 2023-11-28T14:32:06.742 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_blackhole_mds": "false", 2023-11-28T14:32:06.742 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_blackhole_mgr": "false", 2023-11-28T14:32:06.742 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_blackhole_mon": "false", 2023-11-28T14:32:06.742 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_blackhole_osd": "false", 2023-11-28T14:32:06.743 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_client_mode": "crc", 2023-11-28T14:32:06.743 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_cluster_mode": "crc", 2023-11-28T14:32:06.743 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_cluster_type": "", 2023-11-28T14:32:06.743 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_compress_secure": "false", 2023-11-28T14:32:06.743 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_connection_idle_timeout": "900", 2023-11-28T14:32:06.743 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_connection_ready_timeout": "10", 2023-11-28T14:32:06.743 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_crc_data": "true", 2023-11-28T14:32:06.743 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_crc_header": "true", 2023-11-28T14:32:06.743 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_die_on_bad_msg": "false", 2023-11-28T14:32:06.743 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_die_on_bug": "true", 2023-11-28T14:32:06.743 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_die_on_old_message": "true", 2023-11-28T14:32:06.743 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_die_on_skipped_message": "false", 2023-11-28T14:32:06.743 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_die_on_unhandled_msg": "false", 2023-11-28T14:32:06.744 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dispatch_throttle_bytes": "104857600", 2023-11-28T14:32:06.744 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_coremask": "0xF", 2023-11-28T14:32:06.744 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_debug_allow_loopback": "false", 2023-11-28T14:32:06.744 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_devs_allowlist": "", 2023-11-28T14:32:06.744 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_enable_tso": "true", 2023-11-28T14:32:06.744 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2023-11-28T14:32:06.744 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_host_ipv4_addr": "", 2023-11-28T14:32:06.744 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_hugepages": "", 2023-11-28T14:32:06.744 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_hw_flow_control": "true", 2023-11-28T14:32:06.744 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2023-11-28T14:32:06.744 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_lro": "true", 2023-11-28T14:32:06.744 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_memory_channel": "4", 2023-11-28T14:32:06.745 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2023-11-28T14:32:06.745 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_pmd": "", 2023-11-28T14:32:06.745 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_port_id": "0", 2023-11-28T14:32:06.745 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2023-11-28T14:32:06.745 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dump_corrupt_message_level": "1", 2023-11-28T14:32:06.745 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dump_on_send": "false", 2023-11-28T14:32:06.745 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_initial_backoff": "0.200000", 2023-11-28T14:32:06.745 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_inject_delay_max": "1.000000", 2023-11-28T14:32:06.745 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_inject_delay_probability": "0.000000", 2023-11-28T14:32:06.745 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_inject_delay_type": "", 2023-11-28T14:32:06.745 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_inject_internal_delays": "0.000000", 2023-11-28T14:32:06.746 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_inject_network_congestion": "0", 2023-11-28T14:32:06.746 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_inject_socket_failures": "0", 2023-11-28T14:32:06.746 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_learn_addr_from_peer": "true", 2023-11-28T14:32:06.746 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_max_accept_failures": "4", 2023-11-28T14:32:06.746 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_max_backoff": "15.000000", 2023-11-28T14:32:06.746 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_mon_client_mode": "crc", 2023-11-28T14:32:06.746 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_mon_cluster_mode": "crc", 2023-11-28T14:32:06.746 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_mon_service_mode": "crc", 2023-11-28T14:32:06.746 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_osd_compress_min_size": "1024", 2023-11-28T14:32:06.746 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_osd_compress_mode": "none", 2023-11-28T14:32:06.746 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_osd_compression_algorithm": "snappy", 2023-11-28T14:32:06.746 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2023-11-28T14:32:06.747 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_pq_min_cost": "65536", 2023-11-28T14:32:06.747 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_public_type": "", 2023-11-28T14:32:06.747 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_service_mode": "crc", 2023-11-28T14:32:06.747 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_tcp_listen_backlog": "512", 2023-11-28T14:32:06.747 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_tcp_nodelay": "true", 2023-11-28T14:32:06.747 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_tcp_prefetch_max_size": "4096", 2023-11-28T14:32:06.747 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_tcp_rcvbuf": "0", 2023-11-28T14:32:06.747 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_type": "async+posix", 2023-11-28T14:32:06.747 INFO:teuthology.orchestra.run.smithi055.stdout: "no_config_file": "false", 2023-11-28T14:32:06.747 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_completion_locks_per_session": "32", 2023-11-28T14:32:06.747 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_debug_inject_relock_delay": "false", 2023-11-28T14:32:06.748 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_inflight_op_bytes": "104857600", 2023-11-28T14:32:06.748 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_inflight_ops": "1024", 2023-11-28T14:32:06.748 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_inject_no_watch_ping": "false", 2023-11-28T14:32:06.748 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_retry_writes_after_first_reply": "false", 2023-11-28T14:32:06.748 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_tick_interval": "5.000000", 2023-11-28T14:32:06.748 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_timeout": "10.000000", 2023-11-28T14:32:06.748 INFO:teuthology.orchestra.run.smithi055.stdout: "objectstore_blackhole": "false", 2023-11-28T14:32:06.748 INFO:teuthology.orchestra.run.smithi055.stdout: "openssl_engine_opts": "", 2023-11-28T14:32:06.748 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_delay_time": "5.000000", 2023-11-28T14:32:06.748 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_hist_halflife": "1000", 2023-11-28T14:32:06.748 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_max_low_ops": "2", 2023-11-28T14:32:06.748 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_max_ops": "4", 2023-11-28T14:32:06.749 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_min_evict_effort": "0.100000", 2023-11-28T14:32:06.749 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_quantize_effort": "0.100000", 2023-11-28T14:32:06.749 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_slop": "0.020000", 2023-11-28T14:32:06.749 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_aggregated_slow_ops_logging": "true", 2023-11-28T14:32:06.749 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_allow_recovery_below_min_size": "true", 2023-11-28T14:32:06.749 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_asio_thread_count": "2", 2023-11-28T14:32:06.749 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_async_recovery_min_cost": "100", 2023-11-28T14:32:06.749 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_auto_mark_unfound_lost": "false", 2023-11-28T14:32:06.749 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_backfill_retry_interval": "30.000000", 2023-11-28T14:32:06.749 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_backfill_scan_max": "512", 2023-11-28T14:32:06.749 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_backfill_scan_min": "64", 2023-11-28T14:32:06.749 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_backoff_on_degraded": "false", 2023-11-28T14:32:06.750 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_backoff_on_peering": "false", 2023-11-28T14:32:06.750 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_backoff_on_unfound": "true", 2023-11-28T14:32:06.750 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_beacon_report_interval": "300", 2023-11-28T14:32:06.750 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_bench_duration": "30", 2023-11-28T14:32:06.750 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_bench_large_size_max_throughput": "104857600", 2023-11-28T14:32:06.750 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_bench_max_block_size": "67108864", 2023-11-28T14:32:06.750 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_bench_small_size_max_iops": "100", 2023-11-28T14:32:06.750 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_blkin_trace_all": "false", 2023-11-28T14:32:06.750 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_blocked_scrub_grace_period": "120", 2023-11-28T14:32:06.750 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2023-11-28T14:32:06.750 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2023-11-28T14:32:06.751 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2023-11-28T14:32:06.751 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_check_for_log_corruption": "false", 2023-11-28T14:32:06.751 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_check_max_object_name_len_on_startup": "true", 2023-11-28T14:32:06.751 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-11-28T14:32:06.751 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2023-11-28T14:32:06.751 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-11-28T14:32:06.751 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_class_update_on_start": "true", 2023-11-28T14:32:06.751 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_client_message_cap": "256", 2023-11-28T14:32:06.751 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_client_message_size_cap": "524288000", 2023-11-28T14:32:06.751 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_client_op_priority": "63", 2023-11-28T14:32:06.751 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_client_watch_timeout": "30", 2023-11-28T14:32:06.751 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_command_max_records": "256", 2023-11-28T14:32:06.752 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_compact_on_start": "false", 2023-11-28T14:32:06.752 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_copyfrom_max_chunk": "8388608", 2023-11-28T14:32:06.752 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_crush_chooseleaf_type": "0", 2023-11-28T14:32:06.752 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_crush_initial_weight": "-1.000000", 2023-11-28T14:32:06.752 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_crush_update_on_start": "true", 2023-11-28T14:32:06.752 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_crush_update_weight_set": "true", 2023-11-28T14:32:06.752 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2023-11-28T14:32:06.752 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2023-11-28T14:32:06.752 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2023-11-28T14:32:06.752 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_drop_ping_duration": "0", 2023-11-28T14:32:06.752 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_drop_ping_probability": "0.000000", 2023-11-28T14:32:06.752 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_feed_pullee": "-1", 2023-11-28T14:32:06.753 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_inject_copyfrom_error": "false", 2023-11-28T14:32:06.753 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2023-11-28T14:32:06.753 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2023-11-28T14:32:06.753 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_misdirected_ops": "false", 2023-11-28T14:32:06.753 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_no_acting_change": "false", 2023-11-28T14:32:06.753 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_no_purge_strays": "false", 2023-11-28T14:32:06.753 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_op_order": "false", 2023-11-28T14:32:06.753 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_pg_log_writeout": "false", 2023-11-28T14:32:06.753 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_pretend_recovery_active": "false", 2023-11-28T14:32:06.753 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_random_push_read_error": "0.000000", 2023-11-28T14:32:06.753 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2023-11-28T14:32:06.754 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_shutdown": "false", 2023-11-28T14:32:06.754 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2023-11-28T14:32:06.754 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2023-11-28T14:32:06.754 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_trim_objects": "false", 2023-11-28T14:32:06.754 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_verify_cached_snaps": "false", 2023-11-28T14:32:06.754 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_verify_missing_on_start": "false", 2023-11-28T14:32:06.754 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_verify_snaps": "false", 2023-11-28T14:32:06.754 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_verify_stray_on_activate": "false", 2023-11-28T14:32:06.754 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_interval": "604800.000000", 2023-11-28T14:32:06.754 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_keys": "1024", 2023-11-28T14:32:06.754 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2023-11-28T14:32:06.755 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2023-11-28T14:32:06.755 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2023-11-28T14:32:06.755 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_stride": "524288", 2023-11-28T14:32:06.755 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2023-11-28T14:32:06.755 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_default_data_pool_replay_window": "5", 2023-11-28T14:32:06.755 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_default_notify_timeout": "30", 2023-11-28T14:32:06.755 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_delete_sleep": "0.000000", 2023-11-28T14:32:06.755 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_delete_sleep_hdd": "5.000000", 2023-11-28T14:32:06.755 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_delete_sleep_hybrid": "1.000000", 2023-11-28T14:32:06.755 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_delete_sleep_ssd": "1.000000", 2023-11-28T14:32:06.755 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_discard_disconnected_ops": "true", 2023-11-28T14:32:06.755 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_enable_op_tracker": "true", 2023-11-28T14:32:06.756 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2023-11-28T14:32:06.756 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_extblkdev_plugins": "vdo", 2023-11-28T14:32:06.756 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_failsafe_full_ratio": "0.970000", 2023-11-28T14:32:06.756 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_fast_fail_on_connection_refused": "true", 2023-11-28T14:32:06.756 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_fast_info": "true", 2023-11-28T14:32:06.756 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_fast_shutdown": "true", 2023-11-28T14:32:06.756 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_fast_shutdown_notify_mon": "true", 2023-11-28T14:32:06.756 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_fast_shutdown_timeout": "15", 2023-11-28T14:32:06.756 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_find_best_info_ignore_history_les": "false", 2023-11-28T14:32:06.756 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_force_auth_primary_missing_objects": "100", 2023-11-28T14:32:06.756 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2023-11-28T14:32:06.757 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_function_tracing": "false", 2023-11-28T14:32:06.757 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_grace": "20", 2023-11-28T14:32:06.757 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_interval": "6", 2023-11-28T14:32:06.757 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2023-11-28T14:32:06.757 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_min_peers": "10", 2023-11-28T14:32:06.757 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_min_size": "2000", 2023-11-28T14:32:06.757 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_stale": "600", 2023-11-28T14:32:06.757 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2023-11-28T14:32:06.757 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_hit_set_max_size": "100000", 2023-11-28T14:32:06.757 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_hit_set_min_size": "1000", 2023-11-28T14:32:06.757 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_hit_set_namespace": ".ceph-internal", 2023-11-28T14:32:06.757 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_ignore_stale_divergent_priors": "false", 2023-11-28T14:32:06.758 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2023-11-28T14:32:06.758 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_inject_failure_on_pg_removal": "false", 2023-11-28T14:32:06.758 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2023-11-28T14:32:06.758 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_journal_flush_on_shutdown": "true", 2023-11-28T14:32:06.758 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_journal_size": "5120", 2023-11-28T14:32:06.758 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_kill_backfill_at": "0", 2023-11-28T14:32:06.758 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_loop_before_reset_tphandle": "64", 2023-11-28T14:32:06.758 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_map_cache_size": "50", 2023-11-28T14:32:06.758 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_map_dedup": "true", 2023-11-28T14:32:06.758 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_map_message_max": "40", 2023-11-28T14:32:06.758 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_map_message_max_bytes": "10485760", 2023-11-28T14:32:06.759 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_map_share_max_epochs": "40", 2023-11-28T14:32:06.759 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_attr_name_len": "100", 2023-11-28T14:32:06.759 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_attr_size": "0", 2023-11-28T14:32:06.759 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_backfills": "1", 2023-11-28T14:32:06.759 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_markdown_count": "5", 2023-11-28T14:32:06.759 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_markdown_period": "600", 2023-11-28T14:32:06.759 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_object_name_len": "2048", 2023-11-28T14:32:06.759 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_object_namespace_len": "256", 2023-11-28T14:32:06.759 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_object_size": "134217728", 2023-11-28T14:32:06.759 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2023-11-28T14:32:06.759 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_omap_entries_per_request": "1024", 2023-11-28T14:32:06.759 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_pg_blocked_by": "16", 2023-11-28T14:32:06.760 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_pg_log_entries": "10000", 2023-11-28T14:32:06.760 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2023-11-28T14:32:06.760 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_pgls": "1024", 2023-11-28T14:32:06.760 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_push_cost": "8388608", 2023-11-28T14:32:06.760 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_push_objects": "10", 2023-11-28T14:32:06.760 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_scrubs": "3", 2023-11-28T14:32:06.760 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2023-11-28T14:32:06.760 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_trimming_pgs": "2", 2023-11-28T14:32:06.760 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_write_op_reply_len": "64", 2023-11-28T14:32:06.760 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_write_size": "90", 2023-11-28T14:32:06.760 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2023-11-28T14:32:06.761 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2023-11-28T14:32:06.761 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2023-11-28T14:32:06.761 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2023-11-28T14:32:06.761 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2023-11-28T14:32:06.761 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2023-11-28T14:32:06.761 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2023-11-28T14:32:06.761 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_override_recovery_settings": "false", 2023-11-28T14:32:06.761 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_profile": "balanced", 2023-11-28T14:32:06.761 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2023-11-28T14:32:06.761 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2023-11-28T14:32:06.761 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2023-11-28T14:32:06.761 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2023-11-28T14:32:06.762 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2023-11-28T14:32:06.762 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2023-11-28T14:32:06.762 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2023-11-28T14:32:06.762 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2023-11-28T14:32:06.762 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2023-11-28T14:32:06.762 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_client_wgt": "1", 2023-11-28T14:32:06.762 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_skip_benchmark": "false", 2023-11-28T14:32:06.762 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_base": "805306368", 2023-11-28T14:32:06.762 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_cache_min": "134217728", 2023-11-28T14:32:06.762 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_cache_resize_interval": "1.000000", 2023-11-28T14:32:06.763 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_expected_fragmentation": "0.150000", 2023-11-28T14:32:06.763 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_target": "4294967296", 2023-11-28T14:32:06.763 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_target_autotune": "false", 2023-11-28T14:32:06.763 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2023-11-28T14:32:06.763 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_min_pg_log_entries": "250", 2023-11-28T14:32:06.763 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_min_recovery_priority": "0", 2023-11-28T14:32:06.763 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mon_heartbeat_interval": "30", 2023-11-28T14:32:06.763 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2023-11-28T14:32:06.763 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mon_report_interval": "5", 2023-11-28T14:32:06.763 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mon_report_max_in_flight": "2", 2023-11-28T14:32:06.764 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mon_shutdown_timeout": "5.000000", 2023-11-28T14:32:06.764 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_num_cache_shards": "32", 2023-11-28T14:32:06.764 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_num_op_tracker_shard": "32", 2023-11-28T14:32:06.764 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_numa_auto_affinity": "true", 2023-11-28T14:32:06.764 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_numa_node": "-1", 2023-11-28T14:32:06.764 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_numa_prefer_iface": "true", 2023-11-28T14:32:06.764 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_object_clean_region_max_num_intervals": "10", 2023-11-28T14:32:06.764 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_objecter_finishers": "1", 2023-11-28T14:32:06.764 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_objectstore": "bluestore", 2023-11-28T14:32:06.765 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_objectstore_fuse": "false", 2023-11-28T14:32:06.765 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_objectstore_tracing": "false", 2023-11-28T14:32:06.765 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_complaint_time": "30.000000", 2023-11-28T14:32:06.765 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_history_duration": "600", 2023-11-28T14:32:06.765 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_history_size": "20", 2023-11-28T14:32:06.765 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_history_slow_op_size": "20", 2023-11-28T14:32:06.765 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2023-11-28T14:32:06.765 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_log_threshold": "5", 2023-11-28T14:32:06.765 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_num_shards": "0", 2023-11-28T14:32:06.765 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_num_shards_hdd": "5", 2023-11-28T14:32:06.765 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_num_shards_ssd": "8", 2023-11-28T14:32:06.766 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_num_threads_per_shard": "0", 2023-11-28T14:32:06.766 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2023-11-28T14:32:06.766 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2023-11-28T14:32:06.766 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2023-11-28T14:32:06.766 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_pq_min_cost": "65536", 2023-11-28T14:32:06.766 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_queue": "mclock_scheduler", 2023-11-28T14:32:06.766 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_queue_cut_off": "high", 2023-11-28T14:32:06.766 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_thread_suicide_timeout": "150", 2023-11-28T14:32:06.766 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_thread_timeout": "15", 2023-11-28T14:32:06.766 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_open_classes_on_start": "true", 2023-11-28T14:32:06.766 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_os_flags": "0", 2023-11-28T14:32:06.766 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_peering_op_priority": "255", 2023-11-28T14:32:06.767 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_delete_cost": "1048576", 2023-11-28T14:32:06.767 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_delete_priority": "5", 2023-11-28T14:32:06.767 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2023-11-28T14:32:06.767 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2023-11-28T14:32:06.767 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_log_dups_tracked": "3000", 2023-11-28T14:32:06.767 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_log_trim_max": "10000", 2023-11-28T14:32:06.767 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_log_trim_min": "100", 2023-11-28T14:32:06.767 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2023-11-28T14:32:06.767 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_object_context_cache_count": "64", 2023-11-28T14:32:06.767 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_stat_report_interval_max": "500", 2023-11-28T14:32:06.767 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2023-11-28T14:32:06.767 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_cache_min_evict_age": "0", 2023-11-28T14:32:06.768 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_cache_min_flush_age": "0", 2023-11-28T14:32:06.768 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2023-11-28T14:32:06.768 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2023-11-28T14:32:06.768 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2023-11-28T14:32:06.768 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_crimson": "true", 2023-11-28T14:32:06.768 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_crush_rule": "-1", 2023-11-28T14:32:06.768 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_ec_fast_read": "false", 2023-11-28T14:32:06.768 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2023-11-28T14:32:06.768 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_flag_bulk": "false", 2023-11-28T14:32:06.768 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_flag_hashpspool": "true", 2023-11-28T14:32:06.768 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_flag_nodelete": "false", 2023-11-28T14:32:06.768 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_flag_nopgchange": "false", 2023-11-28T14:32:06.769 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_flag_nosizechange": "false", 2023-11-28T14:32:06.769 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_flags": "0", 2023-11-28T14:32:06.769 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2023-11-28T14:32:06.769 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_min_size": "2", 2023-11-28T14:32:06.769 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2023-11-28T14:32:06.769 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_pg_num": "32", 2023-11-28T14:32:06.769 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_pgp_num": "0", 2023-11-28T14:32:06.769 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2023-11-28T14:32:06.769 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_size": "3", 2023-11-28T14:32:06.769 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_type": "replicated", 2023-11-28T14:32:06.769 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2023-11-28T14:32:06.770 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_use_gmt_hitset": "true", 2023-11-28T14:32:06.770 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_push_per_object_cost": "1000", 2023-11-28T14:32:06.770 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_read_ec_check_for_errors": "false", 2023-11-28T14:32:06.770 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recover_clone_overlap": "true", 2023-11-28T14:32:06.770 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recover_clone_overlap_limit": "10", 2023-11-28T14:32:06.770 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_cost": "20971520", 2023-11-28T14:32:06.770 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_delay_start": "0.000000", 2023-11-28T14:32:06.770 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_max_active": "0", 2023-11-28T14:32:06.770 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_max_active_hdd": "3", 2023-11-28T14:32:06.770 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_max_active_ssd": "10", 2023-11-28T14:32:06.770 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_max_chunk": "8388608", 2023-11-28T14:32:06.770 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2023-11-28T14:32:06.771 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_max_single_start": "1", 2023-11-28T14:32:06.771 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_op_priority": "3", 2023-11-28T14:32:06.771 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_op_warn_multiple": "16", 2023-11-28T14:32:06.771 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_priority": "5", 2023-11-28T14:32:06.771 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_retry_interval": "30.000000", 2023-11-28T14:32:06.771 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_sleep": "0.000000", 2023-11-28T14:32:06.771 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_sleep_hdd": "0.100000", 2023-11-28T14:32:06.771 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2023-11-28T14:32:06.771 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_sleep_ssd": "0.000000", 2023-11-28T14:32:06.771 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_repair_during_recovery": "false", 2023-11-28T14:32:06.771 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_requested_scrub_priority": "120", 2023-11-28T14:32:06.771 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2023-11-28T14:32:06.772 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_rollback_to_cluster_snap": "", 2023-11-28T14:32:06.772 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_auto_repair": "false", 2023-11-28T14:32:06.772 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_auto_repair_num_errors": "5", 2023-11-28T14:32:06.772 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_backoff_ratio": "0.660000", 2023-11-28T14:32:06.772 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_begin_hour": "0", 2023-11-28T14:32:06.772 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_begin_week_day": "0", 2023-11-28T14:32:06.772 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_chunk_max": "25", 2023-11-28T14:32:06.772 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_chunk_min": "5", 2023-11-28T14:32:06.772 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_cost": "52428800", 2023-11-28T14:32:06.772 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_during_recovery": "false", 2023-11-28T14:32:06.772 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_end_hour": "0", 2023-11-28T14:32:06.773 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_end_week_day": "0", 2023-11-28T14:32:06.773 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_event_cost": "4096", 2023-11-28T14:32:06.773 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_extended_sleep": "0.000000", 2023-11-28T14:32:06.773 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2023-11-28T14:32:06.773 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_invalid_stats": "true", 2023-11-28T14:32:06.773 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_load_threshold": "0.500000", 2023-11-28T14:32:06.773 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_max_interval": "604800.000000", 2023-11-28T14:32:06.773 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_max_preemptions": "5", 2023-11-28T14:32:06.773 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_min_interval": "86400.000000", 2023-11-28T14:32:06.773 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_priority": "5", 2023-11-28T14:32:06.773 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_reservation_timeout": "300000", 2023-11-28T14:32:06.773 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_sleep": "0.000000", 2023-11-28T14:32:06.774 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_slow_reservation_response": "30000", 2023-11-28T14:32:06.774 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_shallow_scrub_chunk_max": "100", 2023-11-28T14:32:06.774 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_shallow_scrub_chunk_min": "50", 2023-11-28T14:32:06.774 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_shutdown_pgref_assert": "false", 2023-11-28T14:32:06.774 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_skip_data_digest": "false", 2023-11-28T14:32:06.774 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_smart_report_timeout": "5", 2023-11-28T14:32:06.774 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_snap_trim_cost": "1048576", 2023-11-28T14:32:06.774 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_snap_trim_priority": "5", 2023-11-28T14:32:06.774 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_snap_trim_sleep": "0.000000", 2023-11-28T14:32:06.774 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2023-11-28T14:32:06.774 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2023-11-28T14:32:06.775 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2023-11-28T14:32:06.775 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_stats_update_period_not_scrubbing": "120", 2023-11-28T14:32:06.775 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_stats_update_period_scrubbing": "15", 2023-11-28T14:32:06.775 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2023-11-28T14:32:06.775 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_target_transaction_size": "30", 2023-11-28T14:32:06.775 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_hit_set_count": "4", 2023-11-28T14:32:06.775 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2023-11-28T14:32:06.775 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2023-11-28T14:32:06.775 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2023-11-28T14:32:06.775 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2023-11-28T14:32:06.775 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2023-11-28T14:32:06.775 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2023-11-28T14:32:06.776 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_mode": "writeback", 2023-11-28T14:32:06.776 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2023-11-28T14:32:06.776 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_promote_max_objects_sec": "25", 2023-11-28T14:32:06.776 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tracing": "false", 2023-11-28T14:32:06.776 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_use_stale_snap": "false", 2023-11-28T14:32:06.776 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2023-11-28T14:32:06.776 INFO:teuthology.orchestra.run.smithi055.stdout: "osdc_blkin_trace_all": "false", 2023-11-28T14:32:06.776 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_kill_at": "0", 2023-11-28T14:32:06.776 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_max_join_drift": "10", 2023-11-28T14:32:06.776 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_min": "500", 2023-11-28T14:32:06.776 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_min_wait": "0.050000", 2023-11-28T14:32:06.776 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_propose_interval": "1.000000", 2023-11-28T14:32:06.777 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_service_trim_max": "500", 2023-11-28T14:32:06.777 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_service_trim_max_multiplier": "20", 2023-11-28T14:32:06.777 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_service_trim_min": "10", 2023-11-28T14:32:06.777 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_stash_full_interval": "25", 2023-11-28T14:32:06.777 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_trim_max": "500", 2023-11-28T14:32:06.777 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_trim_min": "250", 2023-11-28T14:32:06.777 INFO:teuthology.orchestra.run.smithi055.stdout: "perf": "true", 2023-11-28T14:32:06.777 INFO:teuthology.orchestra.run.smithi055.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2023-11-28T14:32:06.777 INFO:teuthology.orchestra.run.smithi055.stdout: "plugin_crypto_accelerator": "crypto_isal", 2023-11-28T14:32:06.777 INFO:teuthology.orchestra.run.smithi055.stdout: "plugin_dir": "/usr/lib64/ceph", 2023-11-28T14:32:06.777 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "-", 2023-11-28T14:32:06.778 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrv": "", 2023-11-28T14:32:06.778 INFO:teuthology.orchestra.run.smithi055.stdout: "public_bind_addr": "-", 2023-11-28T14:32:06.778 INFO:teuthology.orchestra.run.smithi055.stdout: "public_network": "", 2023-11-28T14:32:06.778 INFO:teuthology.orchestra.run.smithi055.stdout: "public_network_interface": "", 2023-11-28T14:32:06.778 INFO:teuthology.orchestra.run.smithi055.stdout: "qat_compressor_enabled": "false", 2023-11-28T14:32:06.778 INFO:teuthology.orchestra.run.smithi055.stdout: "qat_compressor_session_max_number": "256", 2023-11-28T14:32:06.778 INFO:teuthology.orchestra.run.smithi055.stdout: "rados_mon_op_timeout": "0", 2023-11-28T14:32:06.778 INFO:teuthology.orchestra.run.smithi055.stdout: "rados_osd_op_timeout": "0", 2023-11-28T14:32:06.778 INFO:teuthology.orchestra.run.smithi055.stdout: "rados_tracing": "false", 2023-11-28T14:32:06.778 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_atime_update_interval": "60", 2023-11-28T14:32:06.778 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2023-11-28T14:32:06.778 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_balance_parent_reads": "false", 2023-11-28T14:32:06.779 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_balance_snap_reads": "false", 2023-11-28T14:32:06.779 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_blkin_trace_all": "false", 2023-11-28T14:32:06.779 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_blocklist_expire_seconds": "0", 2023-11-28T14:32:06.779 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_blocklist_on_break_lock": "true", 2023-11-28T14:32:06.779 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache": "true", 2023-11-28T14:32:06.779 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_block_writes_upfront": "false", 2023-11-28T14:32:06.779 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_max_dirty": "25165824", 2023-11-28T14:32:06.779 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_max_dirty_age": "1.000000", 2023-11-28T14:32:06.779 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_max_dirty_object": "0", 2023-11-28T14:32:06.779 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_policy": "writearound", 2023-11-28T14:32:06.779 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_size": "33554432", 2023-11-28T14:32:06.779 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_target_dirty": "16777216", 2023-11-28T14:32:06.781 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_writethrough_until_flush": "true", 2023-11-28T14:32:06.781 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_clone_copy_on_read": "false", 2023-11-28T14:32:06.781 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_compression_hint": "none", 2023-11-28T14:32:06.781 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_concurrent_management_ops": "10", 2023-11-28T14:32:06.781 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_config_pool_override_update_timestamp": "0", 2023-11-28T14:32:06.781 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_clone_format": "auto", 2023-11-28T14:32:06.781 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_data_pool": "", 2023-11-28T14:32:06.781 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_features": "61", 2023-11-28T14:32:06.781 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_format": "2", 2023-11-28T14:32:06.782 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_map_options": "", 2023-11-28T14:32:06.782 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_order": "22", 2023-11-28T14:32:06.782 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_pool": "rbd", 2023-11-28T14:32:06.782 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2023-11-28T14:32:06.782 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_stripe_count": "0", 2023-11-28T14:32:06.782 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_stripe_unit": "0", 2023-11-28T14:32:06.782 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_disable_zero_copy_writes": "true", 2023-11-28T14:32:06.782 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_discard_granularity_bytes": "65536", 2023-11-28T14:32:06.782 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_discard_on_zeroed_write_same": "true", 2023-11-28T14:32:06.782 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_enable_alloc_hint": "true", 2023-11-28T14:32:06.782 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2023-11-28T14:32:06.782 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_io_scheduler": "simple", 2023-11-28T14:32:06.783 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2023-11-28T14:32:06.783 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_commit_age": "5.000000", 2023-11-28T14:32:06.783 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2023-11-28T14:32:06.783 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_max_payload_bytes": "16384", 2023-11-28T14:32:06.783 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_object_flush_age": "0.000000", 2023-11-28T14:32:06.783 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_object_flush_bytes": "1048576", 2023-11-28T14:32:06.783 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_object_flush_interval": "0", 2023-11-28T14:32:06.783 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2023-11-28T14:32:06.783 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2023-11-28T14:32:06.783 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_order": "24", 2023-11-28T14:32:06.783 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_pool": "", 2023-11-28T14:32:06.783 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_splay_width": "4", 2023-11-28T14:32:06.784 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_localize_parent_reads": "false", 2023-11-28T14:32:06.784 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_localize_snap_reads": "false", 2023-11-28T14:32:06.784 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2023-11-28T14:32:06.784 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2023-11-28T14:32:06.784 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2023-11-28T14:32:06.784 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2023-11-28T14:32:06.784 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2023-11-28T14:32:06.784 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2023-11-28T14:32:06.784 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_image_policy_type": "simple", 2023-11-28T14:32:06.784 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2023-11-28T14:32:06.784 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_image_state_check_interval": "30", 2023-11-28T14:32:06.784 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2023-11-28T14:32:06.784 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2023-11-28T14:32:06.785 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2023-11-28T14:32:06.785 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2023-11-28T14:32:06.785 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2023-11-28T14:32:06.785 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_autotune": "true", 2023-11-28T14:32:06.785 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_base": "805306368", 2023-11-28T14:32:06.785 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2023-11-28T14:32:06.785 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_cache_min": "134217728", 2023-11-28T14:32:06.785 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2023-11-28T14:32:06.785 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2023-11-28T14:32:06.785 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_target": "4294967296", 2023-11-28T14:32:06.785 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_perf_stats_prio": "5", 2023-11-28T14:32:06.785 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2023-11-28T14:32:06.786 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2023-11-28T14:32:06.786 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirroring_delete_delay": "0", 2023-11-28T14:32:06.786 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2023-11-28T14:32:06.786 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirroring_replay_delay": "0", 2023-11-28T14:32:06.786 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2023-11-28T14:32:06.786 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2023-11-28T14:32:06.786 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_move_to_trash_on_remove": "false", 2023-11-28T14:32:06.786 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2023-11-28T14:32:06.786 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mtime_update_interval": "60", 2023-11-28T14:32:06.786 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_non_blocking_aio": "true", 2023-11-28T14:32:06.786 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_op_thread_timeout": "60", 2023-11-28T14:32:06.786 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_op_threads": "1", 2023-11-28T14:32:06.787 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_parent_cache_enabled": "false", 2023-11-28T14:32:06.787 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_persistent_cache_mode": "disabled", 2023-11-28T14:32:06.787 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_persistent_cache_path": "/tmp", 2023-11-28T14:32:06.787 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_persistent_cache_size": "1073741824", 2023-11-28T14:32:06.787 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_plugins": "", 2023-11-28T14:32:06.787 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_bps_burst": "0", 2023-11-28T14:32:06.787 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_bps_burst_seconds": "1", 2023-11-28T14:32:06.787 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_bps_limit": "0", 2023-11-28T14:32:06.787 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_exclude_ops": "0", 2023-11-28T14:32:06.787 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_iops_burst": "0", 2023-11-28T14:32:06.787 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_iops_burst_seconds": "1", 2023-11-28T14:32:06.787 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_iops_limit": "0", 2023-11-28T14:32:06.788 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_read_bps_burst": "0", 2023-11-28T14:32:06.788 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2023-11-28T14:32:06.788 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_read_bps_limit": "0", 2023-11-28T14:32:06.788 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_read_iops_burst": "0", 2023-11-28T14:32:06.788 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2023-11-28T14:32:06.788 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_read_iops_limit": "0", 2023-11-28T14:32:06.788 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_schedule_tick_min": "50", 2023-11-28T14:32:06.788 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_write_bps_burst": "0", 2023-11-28T14:32:06.788 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2023-11-28T14:32:06.788 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_write_bps_limit": "0", 2023-11-28T14:32:06.788 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_write_iops_burst": "0", 2023-11-28T14:32:06.788 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2023-11-28T14:32:06.789 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_write_iops_limit": "0", 2023-11-28T14:32:06.789 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_quiesce_notification_attempts": "10", 2023-11-28T14:32:06.789 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_read_from_replica_policy": "default", 2023-11-28T14:32:06.789 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2023-11-28T14:32:06.789 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_readahead_max_bytes": "524288", 2023-11-28T14:32:06.789 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_readahead_trigger_requests": "10", 2023-11-28T14:32:06.789 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_request_timed_out_seconds": "30", 2023-11-28T14:32:06.789 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_skip_partial_discard": "true", 2023-11-28T14:32:06.789 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2023-11-28T14:32:06.789 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_tracing": "false", 2023-11-28T14:32:06.789 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_validate_names": "true", 2023-11-28T14:32:06.789 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_validate_pool": "true", 2023-11-28T14:32:06.790 INFO:teuthology.orchestra.run.smithi055.stdout: "restapi_base_url": "", 2023-11-28T14:32:06.790 INFO:teuthology.orchestra.run.smithi055.stdout: "restapi_log_level": "", 2023-11-28T14:32:06.790 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_acl_grants_max_num": "100", 2023-11-28T14:32:06.790 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_admin_entry": "admin", 2023-11-28T14:32:06.790 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2023-11-28T14:32:06.790 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_backend_store": "rados", 2023-11-28T14:32:06.790 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_barbican_url": "", 2023-11-28T14:32:06.790 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_beast_enable_async": "true", 2023-11-28T14:32:06.790 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_counters_cache": "false", 2023-11-28T14:32:06.790 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_counters_cache_size": "10000", 2023-11-28T14:32:06.790 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2023-11-28T14:32:06.790 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_default_quota_max_size": "-1", 2023-11-28T14:32:06.791 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_index_max_aio": "128", 2023-11-28T14:32:06.791 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2023-11-28T14:32:06.791 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_quota_cache_size": "10000", 2023-11-28T14:32:06.791 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_quota_ttl": "600", 2023-11-28T14:32:06.791 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_sync_spawn_window": "20", 2023-11-28T14:32:06.791 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_cache_enabled": "true", 2023-11-28T14:32:06.791 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_cache_expiry_interval": "900", 2023-11-28T14:32:06.791 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_cache_lru_size": "10000", 2023-11-28T14:32:06.791 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_config_store": "rados", 2023-11-28T14:32:06.791 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_content_length_compat": "false", 2023-11-28T14:32:06.791 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_copy_obj_progress": "true", 2023-11-28T14:32:06.791 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2023-11-28T14:32:06.791 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_cors_rules_max_num": "100", 2023-11-28T14:32:06.792 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_cross_domain_policy": "", 2023-11-28T14:32:06.792 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_default_encryption_key": "", 2023-11-28T14:32:06.792 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_addr": "", 2023-11-28T14:32:06.792 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_ca_path": "", 2023-11-28T14:32:06.792 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_client_cert": "", 2023-11-28T14:32:06.792 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_client_key": "", 2023-11-28T14:32:06.792 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_kms_key_template": "", 2023-11-28T14:32:06.792 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_password": "", 2023-11-28T14:32:06.792 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2023-11-28T14:32:06.792 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_username": "", 2023-11-28T14:32:06.792 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_require_ssl": "true", 2023-11-28T14:32:06.792 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2023-11-28T14:32:06.793 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2023-11-28T14:32:06.793 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_backend": "vault", 2023-11-28T14:32:06.793 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2023-11-28T14:32:06.793 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2023-11-28T14:32:06.793 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2023-11-28T14:32:06.793 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2023-11-28T14:32:06.793 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2023-11-28T14:32:06.793 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2023-11-28T14:32:06.793 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2023-11-28T14:32:06.793 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2023-11-28T14:32:06.793 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2023-11-28T14:32:06.794 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2023-11-28T14:32:06.794 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2023-11-28T14:32:06.794 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_suppress_logs": "true", 2023-11-28T14:32:06.794 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_addr": "", 2023-11-28T14:32:06.794 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_auth": "token", 2023-11-28T14:32:06.794 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_namespace": "", 2023-11-28T14:32:06.794 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_prefix": "", 2023-11-28T14:32:06.794 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_secret_engine": "transit", 2023-11-28T14:32:06.794 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_ssl_cacert": "", 2023-11-28T14:32:06.794 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2023-11-28T14:32:06.794 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2023-11-28T14:32:06.794 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_token_file": "", 2023-11-28T14:32:06.795 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_verify_ssl": "true", 2023-11-28T14:32:06.795 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_curl_buffersize": "524288", 2023-11-28T14:32:06.795 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_curl_low_speed_limit": "1024", 2023-11-28T14:32:06.795 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_curl_low_speed_time": "300", 2023-11-28T14:32:06.795 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_curl_tcp_keepalive": "0", 2023-11-28T14:32:06.795 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_curl_wait_timeout_ms": "1000", 2023-11-28T14:32:06.795 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2023-11-28T14:32:06.795 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2023-11-28T14:32:06.795 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2023-11-28T14:32:06.795 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2023-11-28T14:32:06.795 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_l1_fadvise": "4", 2023-11-28T14:32:06.795 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2023-11-28T14:32:06.796 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_libaio_aio_num": "64", 2023-11-28T14:32:06.796 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_libaio_aio_threads": "20", 2023-11-28T14:32:06.796 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d4n_host": "127.0.0.1", 2023-11-28T14:32:06.796 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d4n_port": "6379", 2023-11-28T14:32:06.796 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2023-11-28T14:32:06.796 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_log_changes_size": "1000", 2023-11-28T14:32:06.796 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_log_num_shards": "128", 2023-11-28T14:32:06.796 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_log_obj_prefix": "data_log", 2023-11-28T14:32:06.796 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_log_window": "30", 2023-11-28T14:32:06.796 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_notify_interval_msec": "0", 2023-11-28T14:32:06.796 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_sync_poll_interval": "20", 2023-11-28T14:32:06.796 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_sync_spawn_window": "20", 2023-11-28T14:32:06.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2023-11-28T14:32:06.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_debug_inject_set_olh_err": "0", 2023-11-28T14:32:06.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_default_data_log_backing": "fifo", 2023-11-28T14:32:06.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_default_realm_info_oid": "default.realm", 2023-11-28T14:32:06.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_default_region_info_oid": "default.region", 2023-11-28T14:32:06.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_default_zone_info_oid": "default.zone", 2023-11-28T14:32:06.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2023-11-28T14:32:06.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_defer_to_bucket_acls": "", 2023-11-28T14:32:06.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_delete_multi_obj_max_num": "1000", 2023-11-28T14:32:06.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_admin_lim": "0.000000", 2023-11-28T14:32:06.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_admin_res": "100.000000", 2023-11-28T14:32:06.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2023-11-28T14:32:06.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_auth_lim": "0.000000", 2023-11-28T14:32:06.798 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_auth_res": "200.000000", 2023-11-28T14:32:06.798 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2023-11-28T14:32:06.798 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_data_lim": "0.000000", 2023-11-28T14:32:06.798 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_data_res": "500.000000", 2023-11-28T14:32:06.798 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_data_wgt": "500.000000", 2023-11-28T14:32:06.798 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2023-11-28T14:32:06.798 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_metadata_res": "500.000000", 2023-11-28T14:32:06.798 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2023-11-28T14:32:06.798 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dns_name": "", 2023-11-28T14:32:06.798 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dns_s3website_name": "", 2023-11-28T14:32:06.798 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dynamic_resharding": "true", 2023-11-28T14:32:06.798 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2023-11-28T14:32:06.799 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_gc_threads": "true", 2023-11-28T14:32:06.799 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_lc_threads": "true", 2023-11-28T14:32:06.799 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_ops_log": "false", 2023-11-28T14:32:06.799 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_quota_threads": "true", 2023-11-28T14:32:06.799 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_static_website": "false", 2023-11-28T14:32:06.799 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_usage_log": "false", 2023-11-28T14:32:06.799 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enforce_swift_acls": "true", 2023-11-28T14:32:06.799 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_exit_timeout_secs": "120", 2023-11-28T14:32:06.799 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_expose_bucket": "false", 2023-11-28T14:32:06.799 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_extended_http_attrs": "", 2023-11-28T14:32:06.799 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_filter": "none", 2023-11-28T14:32:06.799 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2023-11-28T14:32:06.800 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_frontends": "beast port=7480", 2023-11-28T14:32:06.800 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_max_concurrent_io": "10", 2023-11-28T14:32:06.800 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_max_deferred": "50", 2023-11-28T14:32:06.800 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2023-11-28T14:32:06.800 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_max_objs": "32", 2023-11-28T14:32:06.800 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_max_queue_size": "134213632", 2023-11-28T14:32:06.800 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_max_trim_chunk": "16", 2023-11-28T14:32:06.800 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_obj_min_wait": "7200", 2023-11-28T14:32:06.800 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_processor_max_time": "3600", 2023-11-28T14:32:06.800 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_processor_period": "3600", 2023-11-28T14:32:06.800 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_get_obj_max_req_size": "4194304", 2023-11-28T14:32:06.801 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_get_obj_window_size": "16777216", 2023-11-28T14:32:06.801 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_healthcheck_disabling_path": "", 2023-11-28T14:32:06.801 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ignore_get_invalid_range": "false", 2023-11-28T14:32:06.801 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_init_timeout": "300", 2023-11-28T14:32:06.801 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2023-11-28T14:32:06.801 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2023-11-28T14:32:06.801 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_accepted_admin_roles": "", 2023-11-28T14:32:06.801 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_accepted_reader_roles": "", 2023-11-28T14:32:06.801 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2023-11-28T14:32:06.801 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_domain": "", 2023-11-28T14:32:06.801 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_password": "", 2023-11-28T14:32:06.801 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_password_path": "", 2023-11-28T14:32:06.802 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_project": "", 2023-11-28T14:32:06.802 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_tenant": "", 2023-11-28T14:32:06.802 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_token": "", 2023-11-28T14:32:06.802 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_token_path": "", 2023-11-28T14:32:06.802 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_user": "", 2023-11-28T14:32:06.802 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_api_version": "2", 2023-11-28T14:32:06.802 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_barbican_domain": "", 2023-11-28T14:32:06.802 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_barbican_password": "", 2023-11-28T14:32:06.802 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_barbican_project": "", 2023-11-28T14:32:06.802 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_barbican_tenant": "", 2023-11-28T14:32:06.802 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_barbican_user": "", 2023-11-28T14:32:06.802 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2023-11-28T14:32:06.803 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_implicit_tenants": "false", 2023-11-28T14:32:06.803 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2023-11-28T14:32:06.803 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_service_token_enabled": "false", 2023-11-28T14:32:06.803 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_token_cache_size": "10000", 2023-11-28T14:32:06.803 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_url": "", 2023-11-28T14:32:06.803 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_verify_ssl": "true", 2023-11-28T14:32:06.803 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_debug_interval": "-1", 2023-11-28T14:32:06.803 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_lock_max_time": "90", 2023-11-28T14:32:06.803 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_max_objs": "32", 2023-11-28T14:32:06.803 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_max_rules": "1000", 2023-11-28T14:32:06.803 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_max_worker": "3", 2023-11-28T14:32:06.803 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_max_wp_worker": "3", 2023-11-28T14:32:06.804 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_thread_delay": "0", 2023-11-28T14:32:06.804 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2023-11-28T14:32:06.804 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ldap_dnattr": "uid", 2023-11-28T14:32:06.804 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2023-11-28T14:32:06.804 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ldap_searchfilter": "", 2023-11-28T14:32:06.804 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2023-11-28T14:32:06.804 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ldap_uri": "ldaps://", 2023-11-28T14:32:06.804 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2023-11-28T14:32:06.804 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_list_bucket_min_readahead": "1000", 2023-11-28T14:32:06.804 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_list_buckets_max_chunk": "1000", 2023-11-28T14:32:06.804 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_log_http_headers": "", 2023-11-28T14:32:06.804 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_log_nonexistent_bucket": "false", 2023-11-28T14:32:06.805 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2023-11-28T14:32:06.805 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_log_object_name_utc": "false", 2023-11-28T14:32:06.805 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lua_max_memory_per_state": "128000", 2023-11-28T14:32:06.805 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2023-11-28T14:32:06.805 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_attr_name_len": "0", 2023-11-28T14:32:06.805 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_attr_size": "0", 2023-11-28T14:32:06.805 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_attrs_num_in_req": "0", 2023-11-28T14:32:06.805 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_chunk_size": "4194304", 2023-11-28T14:32:06.805 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_concurrent_requests": "1024", 2023-11-28T14:32:06.805 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2023-11-28T14:32:06.805 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_dynamic_shards": "1999", 2023-11-28T14:32:06.805 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_listing_results": "1000", 2023-11-28T14:32:06.806 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_notify_retries": "10", 2023-11-28T14:32:06.806 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_objs_per_shard": "100000", 2023-11-28T14:32:06.806 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_put_param_size": "1048576", 2023-11-28T14:32:06.806 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_put_size": "5368709120", 2023-11-28T14:32:06.806 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_slo_entries": "1000", 2023-11-28T14:32:06.806 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_md_log_max_shards": "64", 2023-11-28T14:32:06.806 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_md_notify_interval_msec": "200", 2023-11-28T14:32:06.806 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_meta_sync_poll_interval": "20", 2023-11-28T14:32:06.806 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_meta_sync_spawn_window": "20", 2023-11-28T14:32:06.806 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_mime_types_file": "/etc/mime.types", 2023-11-28T14:32:06.806 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_mp_lock_max_time": "600", 2023-11-28T14:32:06.807 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_multi_obj_del_max_aio": "16", 2023-11-28T14:32:06.807 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_multipart_min_part_size": "5242880", 2023-11-28T14:32:06.807 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_multipart_part_upload_limit": "10000", 2023-11-28T14:32:06.807 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_fhcache_partitions": "3", 2023-11-28T14:32:06.807 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_fhcache_size": "2017", 2023-11-28T14:32:06.807 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_frontends": "rgw-nfs", 2023-11-28T14:32:06.807 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2023-11-28T14:32:06.807 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_lru_lanes": "5", 2023-11-28T14:32:06.807 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_max_gc": "300", 2023-11-28T14:32:06.807 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_namespace_expire_secs": "300", 2023-11-28T14:32:06.807 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_run_gc_threads": "false", 2023-11-28T14:32:06.807 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_run_lc_threads": "false", 2023-11-28T14:32:06.808 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_run_quota_threads": "false", 2023-11-28T14:32:06.808 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_run_sync_thread": "false", 2023-11-28T14:32:06.808 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_s3_fast_attrs": "false", 2023-11-28T14:32:06.808 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_write_completion_interval_s": "10", 2023-11-28T14:32:06.808 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_num_async_rados_threads": "32", 2023-11-28T14:32:06.808 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_num_control_oids": "8", 2023-11-28T14:32:06.808 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_numa_node": "-1", 2023-11-28T14:32:06.808 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_obj_stripe_size": "4194304", 2023-11-28T14:32:06.808 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_obj_tombstone_cache_size": "1000", 2023-11-28T14:32:06.808 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_objexp_chunk_size": "100", 2023-11-28T14:32:06.808 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_objexp_gc_interval": "600", 2023-11-28T14:32:06.809 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_objexp_hints_num_shards": "127", 2023-11-28T14:32:06.809 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_olh_pending_timeout_sec": "3600", 2023-11-28T14:32:06.809 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_op_thread_suicide_timeout": "0", 2023-11-28T14:32:06.809 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_op_thread_timeout": "600", 2023-11-28T14:32:06.809 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_op_tracing": "false", 2023-11-28T14:32:06.809 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_opa_token": "", 2023-11-28T14:32:06.809 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_opa_url": "", 2023-11-28T14:32:06.809 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_opa_verify_ssl": "true", 2023-11-28T14:32:06.809 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ops_log_data_backlog": "5242880", 2023-11-28T14:32:06.809 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2023-11-28T14:32:06.809 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ops_log_rados": "false", 2023-11-28T14:32:06.809 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ops_log_socket_path": "", 2023-11-28T14:32:06.810 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_override_bucket_index_max_shards": "0", 2023-11-28T14:32:06.810 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2023-11-28T14:32:06.810 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2023-11-28T14:32:06.810 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_period_push_interval": "2.000000", 2023-11-28T14:32:06.810 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_period_push_interval_max": "30.000000", 2023-11-28T14:32:06.810 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_period_root_pool": ".rgw.root", 2023-11-28T14:32:06.810 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_policy_reject_invalid_principals": "true", 2023-11-28T14:32:06.810 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2023-11-28T14:32:06.810 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_posix_cache_lanes": "3", 2023-11-28T14:32:06.810 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_posix_cache_lmdb_count": "3", 2023-11-28T14:32:06.810 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_posix_cache_max_buckets": "100", 2023-11-28T14:32:06.811 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_posix_cache_partitions": "3", 2023-11-28T14:32:06.811 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2023-11-28T14:32:06.811 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_print_continue": "true", 2023-11-28T14:32:06.811 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_print_prohibited_content_length": "false", 2023-11-28T14:32:06.811 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_put_obj_max_window_size": "67108864", 2023-11-28T14:32:06.811 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_put_obj_min_window_size": "16777216", 2023-11-28T14:32:06.811 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2023-11-28T14:32:06.811 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_rados_pool_recovery_priority": "5", 2023-11-28T14:32:06.811 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_rados_tracing": "false", 2023-11-28T14:32:06.811 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_realm": "", 2023-11-28T14:32:06.811 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_realm_id": "", 2023-11-28T14:32:06.811 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_realm_root_pool": ".rgw.root", 2023-11-28T14:32:06.812 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_region": "", 2023-11-28T14:32:06.812 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_region_root_pool": ".rgw.root", 2023-11-28T14:32:06.812 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_relaxed_region_enforcement": "false", 2023-11-28T14:32:06.812 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_relaxed_s3_bucket_names": "false", 2023-11-28T14:32:06.812 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2023-11-28T14:32:06.812 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_request_uri": "", 2023-11-28T14:32:06.812 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_reshard_batch_size": "64", 2023-11-28T14:32:06.812 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_reshard_bucket_lock_duration": "360", 2023-11-28T14:32:06.812 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_reshard_max_aio": "128", 2023-11-28T14:32:06.812 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_reshard_num_logs": "16", 2023-11-28T14:32:06.812 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_reshard_thread_interval": "600", 2023-11-28T14:32:06.812 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_resolve_cname": "false", 2023-11-28T14:32:06.813 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_rest_getusage_op_compat": "false", 2023-11-28T14:32:06.813 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_run_sync_thread": "true", 2023-11-28T14:32:06.813 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_auth_order": "sts, external, local", 2023-11-28T14:32:06.813 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_auth_use_keystone": "false", 2023-11-28T14:32:06.813 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_auth_use_ldap": "false", 2023-11-28T14:32:06.813 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_auth_use_rados": "true", 2023-11-28T14:32:06.813 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_auth_use_sts": "false", 2023-11-28T14:32:06.813 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_client_max_sig_ver": "-1", 2023-11-28T14:32:06.813 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_success_create_obj_status": "0", 2023-11-28T14:32:06.813 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_safe_max_objects_per_shard": "102400", 2023-11-28T14:32:06.813 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_scheduler_type": "throttler", 2023-11-28T14:32:06.814 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_script_uri": "", 2023-11-28T14:32:06.814 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_service_provider_name": "", 2023-11-28T14:32:06.814 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_shard_warning_threshold": "90.000000", 2023-11-28T14:32:06.814 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_client_id": "", 2023-11-28T14:32:06.814 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_client_secret": "", 2023-11-28T14:32:06.814 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_entry": "sts", 2023-11-28T14:32:06.814 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_key": "sts", 2023-11-28T14:32:06.814 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_max_session_duration": "43200", 2023-11-28T14:32:06.814 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_min_session_duration": "900", 2023-11-28T14:32:06.814 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_token_introspection_url": "", 2023-11-28T14:32:06.814 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_account_in_url": "false", 2023-11-28T14:32:06.814 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_auth_entry": "auth", 2023-11-28T14:32:06.815 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_auth_url": "", 2023-11-28T14:32:06.815 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_custom_header": "", 2023-11-28T14:32:06.815 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_enforce_content_length": "false", 2023-11-28T14:32:06.815 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_need_stats": "true", 2023-11-28T14:32:06.815 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_tenant_name": "", 2023-11-28T14:32:06.815 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_token_expiration": "86400", 2023-11-28T14:32:06.815 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_url": "", 2023-11-28T14:32:06.815 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_url_prefix": "swift", 2023-11-28T14:32:06.815 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_versioning_enabled": "false", 2023-11-28T14:32:06.815 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2023-11-28T14:32:06.815 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2023-11-28T14:32:06.815 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_lease_period": "120", 2023-11-28T14:32:06.816 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2023-11-28T14:32:06.816 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_log_trim_interval": "1200", 2023-11-28T14:32:06.816 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_log_trim_max_buckets": "16", 2023-11-28T14:32:06.816 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2023-11-28T14:32:06.816 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2023-11-28T14:32:06.816 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_obj_etag_verify": "false", 2023-11-28T14:32:06.816 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_trace_history_size": "4096", 2023-11-28T14:32:06.816 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_trace_per_node_log_size": "32", 2023-11-28T14:32:06.816 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2023-11-28T14:32:06.816 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_thread_pool_size": "512", 2023-11-28T14:32:06.816 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_topic_persistency_max_retries": "0", 2023-11-28T14:32:06.816 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_topic_persistency_sleep_duration": "0", 2023-11-28T14:32:06.817 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_topic_persistency_time_to_live": "0", 2023-11-28T14:32:06.817 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_comment": "", 2023-11-28T14:32:06.817 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_createby": "", 2023-11-28T14:32:06.817 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_encoding": "", 2023-11-28T14:32:06.817 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_flag": "false", 2023-11-28T14:32:06.817 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_max_size": "5368709120", 2023-11-28T14:32:06.817 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_origin": "", 2023-11-28T14:32:06.817 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_sha_unit": "524288", 2023-11-28T14:32:06.817 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_tracker": "", 2023-11-28T14:32:06.817 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_trust_forwarded_https": "false", 2023-11-28T14:32:06.817 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_usage_log_flush_threshold": "1024", 2023-11-28T14:32:06.818 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_usage_log_tick_interval": "30", 2023-11-28T14:32:06.818 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_usage_max_shards": "32", 2023-11-28T14:32:06.818 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_usage_max_user_shards": "1", 2023-11-28T14:32:06.818 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_use_opa_authz": "false", 2023-11-28T14:32:06.818 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_counters_cache": "false", 2023-11-28T14:32:06.818 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_counters_cache_size": "10000", 2023-11-28T14:32:06.818 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_default_quota_max_objects": "-1", 2023-11-28T14:32:06.818 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_default_quota_max_size": "-1", 2023-11-28T14:32:06.818 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_max_buckets": "1000", 2023-11-28T14:32:06.818 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_policies_max_num": "100", 2023-11-28T14:32:06.818 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2023-11-28T14:32:06.818 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_quota_sync_idle_users": "false", 2023-11-28T14:32:06.819 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_quota_sync_interval": "86400", 2023-11-28T14:32:06.819 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_quota_sync_wait_time": "86400", 2023-11-28T14:32:06.819 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_unique_email": "true", 2023-11-28T14:32:06.819 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_verify_ssl": "true", 2023-11-28T14:32:06.819 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_website_routing_rules_max_num": "50", 2023-11-28T14:32:06.819 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_zone": "", 2023-11-28T14:32:06.819 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_zone_id": "", 2023-11-28T14:32:06.819 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_zone_root_pool": ".rgw.root", 2023-11-28T14:32:06.819 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_zonegroup": "", 2023-11-28T14:32:06.819 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_zonegroup_id": "", 2023-11-28T14:32:06.819 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2023-11-28T14:32:06.819 INFO:teuthology.orchestra.run.smithi055.stdout: "rgwlc_auto_session_clear": "true", 2023-11-28T14:32:06.820 INFO:teuthology.orchestra.run.smithi055.stdout: "rgwlc_skip_bucket_step": "false", 2023-11-28T14:32:06.820 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_block_size": "4096", 2023-11-28T14:32:06.820 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_bloom_bits_per_key": "20", 2023-11-28T14:32:06.820 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2023-11-28T14:32:06.820 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2023-11-28T14:32:06.820 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cache_row_ratio": "0.000000", 2023-11-28T14:32:06.820 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cache_shard_bits": "4", 2023-11-28T14:32:06.820 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cache_size": "536870912", 2023-11-28T14:32:06.820 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cache_type": "binned_lru", 2023-11-28T14:32:06.820 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cf_compact_on_deletion": "true", 2023-11-28T14:32:06.820 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2023-11-28T14:32:06.820 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2023-11-28T14:32:06.821 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_collect_compaction_stats": "false", 2023-11-28T14:32:06.821 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_collect_extended_stats": "false", 2023-11-28T14:32:06.821 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_collect_memory_stats": "false", 2023-11-28T14:32:06.821 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_delete_range_threshold": "1048576", 2023-11-28T14:32:06.821 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_index_type": "binary_search", 2023-11-28T14:32:06.821 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_log_to_ceph_log": "true", 2023-11-28T14:32:06.821 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_metadata_block_size": "4096", 2023-11-28T14:32:06.821 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_partition_filters": "false", 2023-11-28T14:32:06.821 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_perf": "false", 2023-11-28T14:32:06.821 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2023-11-28T14:32:06.821 INFO:teuthology.orchestra.run.smithi055.stdout: "rotating_keys_bootstrap_timeout": "30", 2023-11-28T14:32:06.821 INFO:teuthology.orchestra.run.smithi055.stdout: "rotating_keys_renewal_timeout": "10", 2023-11-28T14:32:06.822 INFO:teuthology.orchestra.run.smithi055.stdout: "run_dir": "/var/run/ceph", 2023-11-28T14:32:06.822 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_block_create": "true", 2023-11-28T14:32:06.822 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_cache_lru_size": "67108864", 2023-11-28T14:32:06.822 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_cbjournal_size": "5368709120", 2023-11-28T14:32:06.822 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_data_delta_based_overwrite": "0", 2023-11-28T14:32:06.822 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_default_max_object_size": "16777216", 2023-11-28T14:32:06.822 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_default_object_metadata_reservation": "16777216", 2023-11-28T14:32:06.822 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_device_size": "53687091200", 2023-11-28T14:32:06.822 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_journal_batch_capacity": "16", 2023-11-28T14:32:06.822 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_journal_batch_flush_size": "16777216", 2023-11-28T14:32:06.822 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2023-11-28T14:32:06.823 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_journal_iodepth_limit": "5", 2023-11-28T14:32:06.823 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_main_device_type": "SSD", 2023-11-28T14:32:06.823 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_max_concurrent_transactions": "8", 2023-11-28T14:32:06.823 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2023-11-28T14:32:06.823 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2023-11-28T14:32:06.823 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2023-11-28T14:32:06.823 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_obj_data_write_amplification": "1.250000", 2023-11-28T14:32:06.823 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_segment_size": "67108864", 2023-11-28T14:32:06.823 INFO:teuthology.orchestra.run.smithi055.stdout: "set_keepcaps": "false", 2023-11-28T14:32:06.823 INFO:teuthology.orchestra.run.smithi055.stdout: "setgroup": "", 2023-11-28T14:32:06.823 INFO:teuthology.orchestra.run.smithi055.stdout: "setuser": "", 2023-11-28T14:32:06.823 INFO:teuthology.orchestra.run.smithi055.stdout: "setuser_match_path": "", 2023-11-28T14:32:06.824 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_misplaced_ratio": "0.050000", 2023-11-28T14:32:06.824 INFO:teuthology.orchestra.run.smithi055.stdout: "thp": "false", 2023-11-28T14:32:06.824 INFO:teuthology.orchestra.run.smithi055.stdout: "threadpool_default_timeout": "60", 2023-11-28T14:32:06.824 INFO:teuthology.orchestra.run.smithi055.stdout: "threadpool_empty_queue_max_wait": "2", 2023-11-28T14:32:06.824 INFO:teuthology.orchestra.run.smithi055.stdout: "throttler_perf_counter": "true" 2023-11-28T14:32:06.824 INFO:teuthology.orchestra.run.smithi055.stdout:} 2023-11-28T14:32:06.826 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2023-11-28T14:32:06.936 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:06.934+0000 7f860691e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:06.939 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:06.938+0000 7f860691e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:06.953 INFO:teuthology.orchestra.run.smithi055.stdout:{} 2023-11-28T14:32:06.953 INFO:teuthology.orchestra.run.smithi055.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-11-28T14:32:06.963 DEBUG:teuthology.orchestra.run.smithi060:> type ceph-objectstore-tool 2023-11-28T14:32:06.981 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-11-28T14:32:06.981 DEBUG:teuthology.orchestra.run.smithi055:> type ceph-objectstore-tool 2023-11-28T14:32:06.995 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-11-28T14:32:06.996 INFO:teuthology.run_tasks:Running task rados... 2023-11-28T14:32:07.007 INFO:tasks.rados:Beginning rados... 2023-11-28T14:32:07.007 DEBUG:teuthology.run_tasks:Unwinding manager rados 2023-11-28T14:32:07.018 INFO:tasks.rados:joining rados 2023-11-28T14:32:07.018 INFO:tasks.thrashosds.thrasher:starting do_thrash 2023-11-28T14:32:07.018 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:32:07.019 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:32:07.019 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:32:07.019 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2023-11-28T14:32:07.019 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:32:07.020 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2023-11-28T14:32:07.020 INFO:tasks.rados:starting run 0 out of 1 2023-11-28T14:32:07.020 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2023-11-28T14:32:07.020 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2023-11-28T14:32:07.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:07.150+0000 7fa95619e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:07.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:07.150+0000 7f6e454ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:07.156 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:07.155+0000 7fa95619e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:07.158 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:07.157+0000 7f6e454ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:07.158 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:07.150+0000 7f4382f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:07.162 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:07.161+0000 7f4382f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:07.355 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:07.355 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:32:07.367 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-11-28T14:32:06.458396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107744,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379616,"statfs":{"total":386547056640,"available":386436726784,"internally_reserved":0,"allocated":430080,"data_stored":1858476,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.791200"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.775877+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:32:02.775877+0000","last_peered":"2023-11-28T14:32:02.775877+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:32:02.775877+0000","last_undegraded":"2023-11-28T14:32:02.775877+0000","last_fullsized":"2023-11-28T14:32:02.775877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-28T14:32:03.677692+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:32:03.677692+0000","last_peered":"2023-11-28T14:32:03.677692+0000","last_clean":"2023-11-28T14:32:03.677692+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:32:03.677692+0000","last_undegraded":"2023-11-28T14:32:03.677692+0000","last_fullsized":"2023-11-28T14:32:03.677692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.734692+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:32:02.734692+0000","last_peered":"2023-11-28T14:32:02.734692+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:32:02.734692+0000","last_undegraded":"2023-11-28T14:32:02.734692+0000","last_fullsized":"2023-11-28T14:32:02.734692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-28T14:32:03.678346+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:32:03.678346+0000","last_peered":"2023-11-28T14:32:03.678346+0000","last_clean":"2023-11-28T14:32:03.678346+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:32:03.678346+0000","last_undegraded":"2023-11-28T14:32:03.678346+0000","last_fullsized":"2023-11-28T14:32:03.678346+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.757066+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:32:02.757066+0000","last_peered":"2023-11-28T14:32:02.757066+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:32:02.757066+0000","last_undegraded":"2023-11-28T14:32:02.757066+0000","last_fullsized":"2023-11-28T14:32:02.757066+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.741833+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:32:02.741833+0000","last_peered":"2023-11-28T14:32:02.741833+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:32:02.741833+0000","last_undegraded":"2023-11-28T14:32:02.741833+0000","last_fullsized":"2023-11-28T14:32:02.741833+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.770616+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:32:02.770616+0000","last_peered":"2023-11-28T14:32:02.770616+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:32:02.770616+0000","last_undegraded":"2023-11-28T14:32:02.770616+0000","last_fullsized":"2023-11-28T14:32:02.770616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.758730+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:32:02.758730+0000","last_peered":"2023-11-28T14:32:02.758730+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:32:02.758730+0000","last_undegraded":"2023-11-28T14:32:02.758730+0000","last_fullsized":"2023-11-28T14:32:02.758730+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-28T14:32:02.649369+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:32:02.649369+0000","last_peered":"2023-11-28T14:32:02.649369+0000","last_clean":"2023-11-28T14:32:02.649369+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:32:02.649369+0000","last_undegraded":"2023-11-28T14:32:02.649369+0000","last_fullsized":"2023-11-28T14:32:02.649369+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26956,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344884,"statfs":{"total":96636764160,"available":96609161216,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":612211,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26884,"kb_used_data":60,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344956,"statfs":{"total":96636764160,"available":96609234944,"internally_reserved":0,"allocated":61440,"data_stored":21843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:32:08.123 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'unique_pool_0' created 2023-11-28T14:32:08.141 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2023-11-28T14:32:08.287 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:08.278+0000 7fdba639e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:08.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:08.289+0000 7fdba639e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:09.142 INFO:teuthology.orchestra.run.smithi055.stderr:enabled application 'rados' on pool 'unique_pool_0' 2023-11-28T14:32:09.232 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:32:09.372 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:09.369+0000 7f5e0ad5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:09.375 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:09.375+0000 7f5e0ad5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:10.160 DEBUG:teuthology.orchestra.run.smithi055:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0 2023-11-28T14:32:10.199 INFO:tasks.rados.rados.0.smithi055.stdout:adding op weight read -> 100 2023-11-28T14:32:10.199 INFO:tasks.rados.rados.0.smithi055.stdout:adding op weight write -> 50 2023-11-28T14:32:10.199 INFO:tasks.rados.rados.0.smithi055.stdout:adding op weight delete -> 50 2023-11-28T14:32:10.199 INFO:tasks.rados.rados.0.smithi055.stdout:adding op weight snap_create -> 50 2023-11-28T14:32:10.199 INFO:tasks.rados.rados.0.smithi055.stdout:adding op weight snap_remove -> 50 2023-11-28T14:32:10.199 INFO:tasks.rados.rados.0.smithi055.stdout:adding op weight write_excl -> 50 2023-11-28T14:32:10.199 INFO:tasks.rados.rados.0.smithi055.stdout:ceph version 18.0.0-7503-g5537419c (5537419ca37a48f0ea28d7872ae1f14ccf1a953b) reef (dev) 2023-11-28T14:32:10.199 INFO:tasks.rados.rados.0.smithi055.stdout:Configuration: 2023-11-28T14:32:10.200 INFO:tasks.rados.rados.0.smithi055.stdout: Number of operations: 4000 2023-11-28T14:32:10.200 INFO:tasks.rados.rados.0.smithi055.stdout: Number of objects: 50 2023-11-28T14:32:10.200 INFO:tasks.rados.rados.0.smithi055.stdout: Max in flight operations: 16 2023-11-28T14:32:10.200 INFO:tasks.rados.rados.0.smithi055.stdout: Object size (in bytes): 4000000 2023-11-28T14:32:10.200 INFO:tasks.rados.rados.0.smithi055.stdout: Write stride min: 400000 2023-11-28T14:32:10.200 INFO:tasks.rados.rados.0.smithi055.stdout: Write stride max: 800000 2023-11-28T14:32:11.155 INFO:tasks.rados.rados.0.smithi055.stdout:1: write initial oid 1 2023-11-28T14:32:11.155 INFO:tasks.rados.rados.0.smithi055.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2023-11-28T14:32:11.159 INFO:tasks.rados.rados.0.smithi055.stdout:1: writing smithi05583006-1 from 416807 to 892056 tid 1 2023-11-28T14:32:11.166 INFO:tasks.rados.rados.0.smithi055.stdout:1: writing smithi05583006-1 from 1542129 to 2016807 tid 2 2023-11-28T14:32:11.166 INFO:tasks.rados.rados.0.smithi055.stdout:2: write initial oid 2 2023-11-28T14:32:11.166 INFO:tasks.rados.rados.0.smithi055.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2023-11-28T14:32:11.171 INFO:tasks.rados.rados.0.smithi055.stdout:2: writing smithi05583006-2 from 416807 to 892056 tid 1 2023-11-28T14:32:11.179 INFO:tasks.rados.rados.0.smithi055.stdout:2: writing smithi05583006-2 from 1542129 to 2016807 tid 2 2023-11-28T14:32:11.179 INFO:tasks.rados.rados.0.smithi055.stdout:3: write initial oid 3 2023-11-28T14:32:11.179 INFO:tasks.rados.rados.0.smithi055.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2023-11-28T14:32:11.185 INFO:tasks.rados.rados.0.smithi055.stdout:3: writing smithi05583006-3 from 433614 to 984112 tid 1 2023-11-28T14:32:11.192 INFO:tasks.rados.rados.0.smithi055.stdout:3: writing smithi05583006-3 from 1600611 to 2033614 tid 2 2023-11-28T14:32:11.192 INFO:tasks.rados.rados.0.smithi055.stdout:4: write initial oid 4 2023-11-28T14:32:11.192 INFO:tasks.rados.rados.0.smithi055.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2023-11-28T14:32:11.198 INFO:tasks.rados.rados.0.smithi055.stdout:4: writing smithi05583006-4 from 450421 to 1076168 tid 1 2023-11-28T14:32:11.204 INFO:tasks.rados.rados.0.smithi055.stdout:4: writing smithi05583006-4 from 1659093 to 2050421 tid 2 2023-11-28T14:32:11.204 INFO:tasks.rados.rados.0.smithi055.stdout:5: write initial oid 5 2023-11-28T14:32:11.204 INFO:tasks.rados.rados.0.smithi055.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2023-11-28T14:32:11.212 INFO:tasks.rados.rados.0.smithi055.stdout:5: writing smithi05583006-5 from 467228 to 1168224 tid 1 2023-11-28T14:32:11.219 INFO:tasks.rados.rados.0.smithi055.stdout:5: writing smithi05583006-5 from 1717575 to 2067228 tid 2 2023-11-28T14:32:11.219 INFO:tasks.rados.rados.0.smithi055.stdout:6: write initial oid 6 2023-11-28T14:32:11.219 INFO:tasks.rados.rados.0.smithi055.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2023-11-28T14:32:11.227 INFO:tasks.rados.rados.0.smithi055.stdout:6: writing smithi05583006-6 from 484035 to 1260280 tid 1 2023-11-28T14:32:11.232 INFO:tasks.rados.rados.0.smithi055.stdout:6: writing smithi05583006-6 from 2059704 to 2084035 tid 2 2023-11-28T14:32:11.232 INFO:tasks.rados.rados.0.smithi055.stdout:7: write initial oid 7 2023-11-28T14:32:11.233 INFO:tasks.rados.rados.0.smithi055.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2023-11-28T14:32:11.238 INFO:tasks.rados.rados.0.smithi055.stdout:7: writing smithi05583006-7 from 500842 to 952336 tid 1 2023-11-28T14:32:11.245 INFO:tasks.rados.rados.0.smithi055.stdout:7: writing smithi05583006-7 from 1718186 to 2100842 tid 2 2023-11-28T14:32:11.245 INFO:tasks.rados.rados.0.smithi055.stdout:8: write initial oid 8 2023-11-28T14:32:11.245 INFO:tasks.rados.rados.0.smithi055.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2023-11-28T14:32:11.251 INFO:tasks.rados.rados.0.smithi055.stdout:8: writing smithi05583006-8 from 517649 to 1044392 tid 1 2023-11-28T14:32:11.257 INFO:tasks.rados.rados.0.smithi055.stdout:8: writing smithi05583006-8 from 1776668 to 2117649 tid 2 2023-11-28T14:32:11.257 INFO:tasks.rados.rados.0.smithi055.stdout:9: write initial oid 9 2023-11-28T14:32:11.258 INFO:tasks.rados.rados.0.smithi055.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2023-11-28T14:32:11.265 INFO:tasks.rados.rados.0.smithi055.stdout:9: writing smithi05583006-9 from 534456 to 1252801 tid 1 2023-11-28T14:32:11.270 INFO:tasks.rados.rados.0.smithi055.stdout:9: writing smithi05583006-9 from 1951503 to 2134456 tid 2 2023-11-28T14:32:11.270 INFO:tasks.rados.rados.0.smithi055.stdout:10: write initial oid 10 2023-11-28T14:32:11.270 INFO:tasks.rados.rados.0.smithi055.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2023-11-28T14:32:11.277 INFO:tasks.rados.rados.0.smithi055.stdout:10: writing smithi05583006-10 from 551263 to 1344857 tid 1 2023-11-28T14:32:11.282 INFO:tasks.rados.rados.0.smithi055.stdout:10: writing smithi05583006-10 from 1893632 to 2151263 tid 2 2023-11-28T14:32:11.282 INFO:tasks.rados.rados.0.smithi055.stdout:11: write initial oid 11 2023-11-28T14:32:11.282 INFO:tasks.rados.rados.0.smithi055.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2023-11-28T14:32:11.287 INFO:tasks.rados.rados.0.smithi055.stdout:11: writing smithi05583006-11 from 568070 to 1036913 tid 1 2023-11-28T14:32:11.295 INFO:tasks.rados.rados.0.smithi055.stdout:11: writing smithi05583006-11 from 1552114 to 2168070 tid 2 2023-11-28T14:32:11.295 INFO:tasks.rados.rados.0.smithi055.stdout:12: write initial oid 12 2023-11-28T14:32:11.295 INFO:tasks.rados.rados.0.smithi055.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2023-11-28T14:32:11.301 INFO:tasks.rados.rados.0.smithi055.stdout:12: writing smithi05583006-12 from 584877 to 1128969 tid 1 2023-11-28T14:32:11.308 INFO:tasks.rados.rados.0.smithi055.stdout:12: writing smithi05583006-12 from 1610596 to 2172599 tid 2 2023-11-28T14:32:11.308 INFO:tasks.rados.rados.0.smithi055.stdout:13: write initial oid 13 2023-11-28T14:32:11.308 INFO:tasks.rados.rados.0.smithi055.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2023-11-28T14:32:11.316 INFO:tasks.rados.rados.0.smithi055.stdout:13: writing smithi05583006-13 from 601684 to 1221025 tid 1 2023-11-28T14:32:11.322 INFO:tasks.rados.rados.0.smithi055.stdout:13: writing smithi05583006-13 from 1669078 to 2201684 tid 2 2023-11-28T14:32:11.322 INFO:tasks.rados.rados.0.smithi055.stdout:14: write initial oid 14 2023-11-28T14:32:11.322 INFO:tasks.rados.rados.0.smithi055.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2023-11-28T14:32:11.329 INFO:tasks.rados.rados.0.smithi055.stdout:14: writing smithi05583006-14 from 618491 to 1313081 tid 1 2023-11-28T14:32:11.335 INFO:tasks.rados.rados.0.smithi055.stdout:14: writing smithi05583006-14 from 2011207 to 2218491 tid 2 2023-11-28T14:32:11.335 INFO:tasks.rados.rados.0.smithi055.stdout:15: write initial oid 15 2023-11-28T14:32:11.335 INFO:tasks.rados.rados.0.smithi055.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2023-11-28T14:32:11.344 INFO:tasks.rados.rados.0.smithi055.stdout:15: writing smithi05583006-15 from 635298 to 1405137 tid 1 2023-11-28T14:32:11.349 INFO:tasks.rados.rados.0.smithi055.stdout:15: writing smithi05583006-15 from 2069689 to 2235298 tid 2 2023-11-28T14:32:11.349 INFO:tasks.rados.rados.0.smithi055.stdout:16: write initial oid 16 2023-11-28T14:32:11.349 INFO:tasks.rados.rados.0.smithi055.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2023-11-28T14:32:11.355 INFO:tasks.rados.rados.0.smithi055.stdout:16: writing smithi05583006-16 from 652105 to 1097193 tid 1 2023-11-28T14:32:11.362 INFO:tasks.rados.rados.0.smithi055.stdout:16: writing smithi05583006-16 from 1728171 to 2181159 tid 2 2023-11-28T14:32:11.363 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.363 INFO:tasks.rados.rados.0.smithi055.stdout:2: finishing write tid 1 to smithi05583006-2 2023-11-28T14:32:11.363 INFO:tasks.rados.rados.0.smithi055.stdout:1: finishing write tid 1 to smithi05583006-1 2023-11-28T14:32:11.363 INFO:tasks.rados.rados.0.smithi055.stdout:4: finishing write tid 1 to smithi05583006-4 2023-11-28T14:32:11.363 INFO:tasks.rados.rados.0.smithi055.stdout:1: finishing write tid 2 to smithi05583006-1 2023-11-28T14:32:11.364 INFO:tasks.rados.rados.0.smithi055.stdout:5: finishing write tid 1 to smithi05583006-5 2023-11-28T14:32:11.364 INFO:tasks.rados.rados.0.smithi055.stdout:2: finishing write tid 2 to smithi05583006-2 2023-11-28T14:32:11.364 INFO:tasks.rados.rados.0.smithi055.stdout:1: finishing write tid 3 to smithi05583006-1 2023-11-28T14:32:11.364 INFO:tasks.rados.rados.0.smithi055.stdout:1: finishing write tid 4 to smithi05583006-1 2023-11-28T14:32:11.364 INFO:tasks.rados.rados.0.smithi055.stdout:1: oid 1 updating version 0 to 1 2023-11-28T14:32:11.365 INFO:tasks.rados.rados.0.smithi055.stdout:1: oid 1 updating version 1 to 2 2023-11-28T14:32:11.365 INFO:tasks.rados.rados.0.smithi055.stdout:1: oid 1 updating version 2 to 3 2023-11-28T14:32:11.365 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2023-11-28T14:32:11.365 INFO:tasks.rados.rados.0.smithi055.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2023-11-28T14:32:11.365 INFO:tasks.rados.rados.0.smithi055.stdout:4: finishing write tid 2 to smithi05583006-4 2023-11-28T14:32:11.365 INFO:tasks.rados.rados.0.smithi055.stdout:1: done (15 left) 2023-11-28T14:32:11.366 INFO:tasks.rados.rados.0.smithi055.stdout:17: write initial oid 17 2023-11-28T14:32:11.366 INFO:tasks.rados.rados.0.smithi055.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2023-11-28T14:32:11.370 INFO:tasks.rados.rados.0.smithi055.stdout:17: writing smithi05583006-17 from 668912 to 1305602 tid 1 2023-11-28T14:32:11.376 INFO:tasks.rados.rados.0.smithi055.stdout:17: writing smithi05583006-17 from 1903006 to 2268912 tid 2 2023-11-28T14:32:11.376 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.376 INFO:tasks.rados.rados.0.smithi055.stdout:2: finishing write tid 3 to smithi05583006-2 2023-11-28T14:32:11.376 INFO:tasks.rados.rados.0.smithi055.stdout:2: finishing write tid 4 to smithi05583006-2 2023-11-28T14:32:11.376 INFO:tasks.rados.rados.0.smithi055.stdout:2: oid 2 updating version 0 to 3 2023-11-28T14:32:11.376 INFO:tasks.rados.rados.0.smithi055.stdout:2: oid 2 version 3 is already newer than 2 2023-11-28T14:32:11.376 INFO:tasks.rados.rados.0.smithi055.stdout:2: oid 2 version 3 is already newer than 1 2023-11-28T14:32:11.376 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2023-11-28T14:32:11.376 INFO:tasks.rados.rados.0.smithi055.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2023-11-28T14:32:11.376 INFO:tasks.rados.rados.0.smithi055.stdout:5: finishing write tid 2 to smithi05583006-5 2023-11-28T14:32:11.376 INFO:tasks.rados.rados.0.smithi055.stdout:6: finishing write tid 1 to smithi05583006-6 2023-11-28T14:32:11.377 INFO:tasks.rados.rados.0.smithi055.stdout:2: done (15 left) 2023-11-28T14:32:11.377 INFO:tasks.rados.rados.0.smithi055.stdout:18: write initial oid 18 2023-11-28T14:32:11.377 INFO:tasks.rados.rados.0.smithi055.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2023-11-28T14:32:11.383 INFO:tasks.rados.rados.0.smithi055.stdout:18: writing smithi05583006-18 from 685719 to 1397658 tid 1 2023-11-28T14:32:11.389 INFO:tasks.rados.rados.0.smithi055.stdout:18: writing smithi05583006-18 from 1845135 to 2285719 tid 2 2023-11-28T14:32:11.389 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.389 INFO:tasks.rados.rados.0.smithi055.stdout:3: finishing write tid 1 to smithi05583006-3 2023-11-28T14:32:11.389 INFO:tasks.rados.rados.0.smithi055.stdout:4: finishing write tid 3 to smithi05583006-4 2023-11-28T14:32:11.389 INFO:tasks.rados.rados.0.smithi055.stdout:4: finishing write tid 4 to smithi05583006-4 2023-11-28T14:32:11.389 INFO:tasks.rados.rados.0.smithi055.stdout:4: oid 4 updating version 0 to 2 2023-11-28T14:32:11.389 INFO:tasks.rados.rados.0.smithi055.stdout:4: oid 4 version 2 is already newer than 1 2023-11-28T14:32:11.389 INFO:tasks.rados.rados.0.smithi055.stdout:4: oid 4 updating version 2 to 3 2023-11-28T14:32:11.389 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2023-11-28T14:32:11.389 INFO:tasks.rados.rados.0.smithi055.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2023-11-28T14:32:11.389 INFO:tasks.rados.rados.0.smithi055.stdout:7: finishing write tid 1 to smithi05583006-7 2023-11-28T14:32:11.389 INFO:tasks.rados.rados.0.smithi055.stdout:12: finishing write tid 1 to smithi05583006-12 2023-11-28T14:32:11.390 INFO:tasks.rados.rados.0.smithi055.stdout:5: finishing write tid 3 to smithi05583006-5 2023-11-28T14:32:11.390 INFO:tasks.rados.rados.0.smithi055.stdout:5: finishing write tid 4 to smithi05583006-5 2023-11-28T14:32:11.390 INFO:tasks.rados.rados.0.smithi055.stdout:5: oid 5 updating version 0 to 3 2023-11-28T14:32:11.390 INFO:tasks.rados.rados.0.smithi055.stdout:5: oid 5 version 3 is already newer than 1 2023-11-28T14:32:11.390 INFO:tasks.rados.rados.0.smithi055.stdout:5: oid 5 version 3 is already newer than 2 2023-11-28T14:32:11.390 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2023-11-28T14:32:11.390 INFO:tasks.rados.rados.0.smithi055.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2023-11-28T14:32:11.390 INFO:tasks.rados.rados.0.smithi055.stdout:12: finishing write tid 2 to smithi05583006-12 2023-11-28T14:32:11.390 INFO:tasks.rados.rados.0.smithi055.stdout:4: done (15 left) 2023-11-28T14:32:11.390 INFO:tasks.rados.rados.0.smithi055.stdout:5: done (14 left) 2023-11-28T14:32:11.390 INFO:tasks.rados.rados.0.smithi055.stdout:19: write initial oid 19 2023-11-28T14:32:11.390 INFO:tasks.rados.rados.0.smithi055.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2023-11-28T14:32:11.397 INFO:tasks.rados.rados.0.smithi055.stdout:19: writing smithi05583006-19 from 702526 to 1489714 tid 1 2023-11-28T14:32:11.401 INFO:tasks.rados.rados.0.smithi055.stdout:19: writing smithi05583006-19 from 1903617 to 2302526 tid 2 2023-11-28T14:32:11.401 INFO:tasks.rados.rados.0.smithi055.stdout:20: write initial oid 20 2023-11-28T14:32:11.402 INFO:tasks.rados.rados.0.smithi055.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2023-11-28T14:32:11.408 INFO:tasks.rados.rados.0.smithi055.stdout:20: writing smithi05583006-20 from 719333 to 1181770 tid 1 2023-11-28T14:32:11.415 INFO:tasks.rados.rados.0.smithi055.stdout:20: writing smithi05583006-20 from 1962099 to 2319333 tid 2 2023-11-28T14:32:11.415 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.415 INFO:tasks.rados.rados.0.smithi055.stdout:9: finishing write tid 1 to smithi05583006-9 2023-11-28T14:32:11.415 INFO:tasks.rados.rados.0.smithi055.stdout:7: finishing write tid 2 to smithi05583006-7 2023-11-28T14:32:11.415 INFO:tasks.rados.rados.0.smithi055.stdout:6: finishing write tid 2 to smithi05583006-6 2023-11-28T14:32:11.415 INFO:tasks.rados.rados.0.smithi055.stdout:6: finishing write tid 3 to smithi05583006-6 2023-11-28T14:32:11.415 INFO:tasks.rados.rados.0.smithi055.stdout:3: finishing write tid 2 to smithi05583006-3 2023-11-28T14:32:11.415 INFO:tasks.rados.rados.0.smithi055.stdout:10: finishing write tid 1 to smithi05583006-10 2023-11-28T14:32:11.415 INFO:tasks.rados.rados.0.smithi055.stdout:6: finishing write tid 4 to smithi05583006-6 2023-11-28T14:32:11.416 INFO:tasks.rados.rados.0.smithi055.stdout:6: oid 6 updating version 0 to 1 2023-11-28T14:32:11.416 INFO:tasks.rados.rados.0.smithi055.stdout:6: oid 6 updating version 1 to 3 2023-11-28T14:32:11.416 INFO:tasks.rados.rados.0.smithi055.stdout:6: oid 6 version 3 is already newer than 3 2023-11-28T14:32:11.416 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2023-11-28T14:32:11.416 INFO:tasks.rados.rados.0.smithi055.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2023-11-28T14:32:11.416 INFO:tasks.rados.rados.0.smithi055.stdout:7: finishing write tid 3 to smithi05583006-7 2023-11-28T14:32:11.416 INFO:tasks.rados.rados.0.smithi055.stdout:6: done (15 left) 2023-11-28T14:32:11.416 INFO:tasks.rados.rados.0.smithi055.stdout:21: write initial oid 21 2023-11-28T14:32:11.416 INFO:tasks.rados.rados.0.smithi055.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2023-11-28T14:32:11.421 INFO:tasks.rados.rados.0.smithi055.stdout:21: writing smithi05583006-21 from 736140 to 1273826 tid 1 2023-11-28T14:32:11.427 INFO:tasks.rados.rados.0.smithi055.stdout:21: writing smithi05583006-21 from 2020581 to 2336140 tid 2 2023-11-28T14:32:11.427 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.430 INFO:tasks.rados.rados.0.smithi055.stdout:7: finishing write tid 4 to smithi05583006-7 2023-11-28T14:32:11.430 INFO:tasks.rados.rados.0.smithi055.stdout:7: oid 7 updating version 0 to 3 2023-11-28T14:32:11.430 INFO:tasks.rados.rados.0.smithi055.stdout:7: oid 7 version 3 is already newer than 3 2023-11-28T14:32:11.430 INFO:tasks.rados.rados.0.smithi055.stdout:7: oid 7 version 3 is already newer than 1 2023-11-28T14:32:11.430 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-11-28T14:32:11.430 INFO:tasks.rados.rados.0.smithi055.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2023-11-28T14:32:11.430 INFO:tasks.rados.rados.0.smithi055.stdout:7: done (15 left) 2023-11-28T14:32:11.431 INFO:tasks.rados.rados.0.smithi055.stdout:22: write initial oid 22 2023-11-28T14:32:11.431 INFO:tasks.rados.rados.0.smithi055.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2023-11-28T14:32:11.437 INFO:tasks.rados.rados.0.smithi055.stdout:22: writing smithi05583006-22 from 752947 to 1365882 tid 1 2023-11-28T14:32:11.443 INFO:tasks.rados.rados.0.smithi055.stdout:22: writing smithi05583006-22 from 1962710 to 2352947 tid 2 2023-11-28T14:32:11.443 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.443 INFO:tasks.rados.rados.0.smithi055.stdout:12: finishing write tid 3 to smithi05583006-12 2023-11-28T14:32:11.443 INFO:tasks.rados.rados.0.smithi055.stdout:12: finishing write tid 4 to smithi05583006-12 2023-11-28T14:32:11.443 INFO:tasks.rados.rados.0.smithi055.stdout:12: oid 12 updating version 0 to 2 2023-11-28T14:32:11.443 INFO:tasks.rados.rados.0.smithi055.stdout:12: oid 12 version 2 is already newer than 1 2023-11-28T14:32:11.443 INFO:tasks.rados.rados.0.smithi055.stdout:12: oid 12 updating version 2 to 3 2023-11-28T14:32:11.444 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2023-11-28T14:32:11.444 INFO:tasks.rados.rados.0.smithi055.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2023-11-28T14:32:11.444 INFO:tasks.rados.rados.0.smithi055.stdout:12: done (15 left) 2023-11-28T14:32:11.444 INFO:tasks.rados.rados.0.smithi055.stdout:23: write initial oid 23 2023-11-28T14:32:11.444 INFO:tasks.rados.rados.0.smithi055.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2023-11-28T14:32:11.451 INFO:tasks.rados.rados.0.smithi055.stdout:23: writing smithi05583006-23 from 769754 to 1457938 tid 1 2023-11-28T14:32:11.457 INFO:tasks.rados.rados.0.smithi055.stdout:23: writing smithi05583006-23 from 2021192 to 2369754 tid 2 2023-11-28T14:32:11.457 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.457 INFO:tasks.rados.rados.0.smithi055.stdout:8: finishing write tid 1 to smithi05583006-8 2023-11-28T14:32:11.457 INFO:tasks.rados.rados.0.smithi055.stdout:17: finishing write tid 1 to smithi05583006-17 2023-11-28T14:32:11.457 INFO:tasks.rados.rados.0.smithi055.stdout:15: finishing write tid 1 to smithi05583006-15 2023-11-28T14:32:11.471 INFO:tasks.rados.rados.0.smithi055.stdout:20: finishing write tid 1 to smithi05583006-20 2023-11-28T14:32:11.474 INFO:tasks.rados.rados.0.smithi055.stdout:10: finishing write tid 2 to smithi05583006-10 2023-11-28T14:32:11.476 INFO:tasks.rados.rados.0.smithi055.stdout:3: finishing write tid 3 to smithi05583006-3 2023-11-28T14:32:11.478 INFO:tasks.rados.rados.0.smithi055.stdout:3: finishing write tid 4 to smithi05583006-3 2023-11-28T14:32:11.478 INFO:tasks.rados.rados.0.smithi055.stdout:3: oid 3 updating version 0 to 4 2023-11-28T14:32:11.478 INFO:tasks.rados.rados.0.smithi055.stdout:3: oid 3 updating version 4 to 5 2023-11-28T14:32:11.478 INFO:tasks.rados.rados.0.smithi055.stdout:3: oid 3 updating version 5 to 6 2023-11-28T14:32:11.478 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 3 v 6 (ObjNum 2 snap 0 seq_num 2) dirty exists 2023-11-28T14:32:11.478 INFO:tasks.rados.rados.0.smithi055.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2023-11-28T14:32:11.478 INFO:tasks.rados.rados.0.smithi055.stdout:3: done (15 left) 2023-11-28T14:32:11.478 INFO:tasks.rados.rados.0.smithi055.stdout:24: write initial oid 24 2023-11-28T14:32:11.479 INFO:tasks.rados.rados.0.smithi055.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2023-11-28T14:32:11.484 INFO:tasks.rados.rados.0.smithi055.stdout:24: writing smithi05583006-24 from 786561 to 1266347 tid 1 2023-11-28T14:32:11.491 INFO:tasks.rados.rados.0.smithi055.stdout:24: writing smithi05583006-24 from 1796027 to 2263691 tid 2 2023-11-28T14:32:11.491 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.491 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:32:11.517 INFO:tasks.rados.rados.0.smithi055.stdout:15: finishing write tid 2 to smithi05583006-15 2023-11-28T14:32:11.521 INFO:tasks.rados.rados.0.smithi055.stdout:21: finishing write tid 1 to smithi05583006-21 2023-11-28T14:32:11.521 INFO:tasks.rados.rados.0.smithi055.stdout:9: finishing write tid 2 to smithi05583006-9 2023-11-28T14:32:11.523 INFO:tasks.rados.rados.0.smithi055.stdout:17: finishing write tid 2 to smithi05583006-17 2023-11-28T14:32:11.526 INFO:tasks.rados.rados.0.smithi055.stdout:20: finishing write tid 2 to smithi05583006-20 2023-11-28T14:32:11.530 INFO:tasks.rados.rados.0.smithi055.stdout:8: finishing write tid 2 to smithi05583006-8 2023-11-28T14:32:11.536 INFO:tasks.rados.rados.0.smithi055.stdout:15: finishing write tid 3 to smithi05583006-15 2023-11-28T14:32:11.540 INFO:tasks.rados.rados.0.smithi055.stdout:8: finishing write tid 3 to smithi05583006-8 2023-11-28T14:32:11.542 INFO:tasks.rados.rados.0.smithi055.stdout:8: finishing write tid 4 to smithi05583006-8 2023-11-28T14:32:11.542 INFO:tasks.rados.rados.0.smithi055.stdout:8: oid 8 updating version 0 to 6 2023-11-28T14:32:11.542 INFO:tasks.rados.rados.0.smithi055.stdout:8: oid 8 version 6 is already newer than 4 2023-11-28T14:32:11.542 INFO:tasks.rados.rados.0.smithi055.stdout:8: oid 8 version 6 is already newer than 6 2023-11-28T14:32:11.543 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2023-11-28T14:32:11.543 INFO:tasks.rados.rados.0.smithi055.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2023-11-28T14:32:11.543 INFO:tasks.rados.rados.0.smithi055.stdout:8: done (15 left) 2023-11-28T14:32:11.543 INFO:tasks.rados.rados.0.smithi055.stdout:25: write initial oid 25 2023-11-28T14:32:11.543 INFO:tasks.rados.rados.0.smithi055.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2023-11-28T14:32:11.547 INFO:tasks.rados.rados.0.smithi055.stdout:25: writing smithi05583006-25 from 403368 to 958403 tid 1 2023-11-28T14:32:11.556 INFO:tasks.rados.rados.0.smithi055.stdout:25: writing smithi05583006-25 from 1454509 to 2182184 tid 2 2023-11-28T14:32:11.556 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.556 INFO:tasks.rados.rados.0.smithi055.stdout:24: finishing write tid 1 to smithi05583006-24 2023-11-28T14:32:11.556 INFO:tasks.rados.rados.0.smithi055.stdout:15: finishing write tid 4 to smithi05583006-15 2023-11-28T14:32:11.556 INFO:tasks.rados.rados.0.smithi055.stdout:15: oid 15 updating version 0 to 3 2023-11-28T14:32:11.556 INFO:tasks.rados.rados.0.smithi055.stdout:15: oid 15 version 3 is already newer than 3 2023-11-28T14:32:11.556 INFO:tasks.rados.rados.0.smithi055.stdout:15: oid 15 version 3 is already newer than 1 2023-11-28T14:32:11.556 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2023-11-28T14:32:11.557 INFO:tasks.rados.rados.0.smithi055.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2023-11-28T14:32:11.557 INFO:tasks.rados.rados.0.smithi055.stdout:15: done (15 left) 2023-11-28T14:32:11.557 INFO:tasks.rados.rados.0.smithi055.stdout:26: write initial oid 26 2023-11-28T14:32:11.557 INFO:tasks.rados.rados.0.smithi055.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2023-11-28T14:32:11.562 INFO:tasks.rados.rados.0.smithi055.stdout:26: writing smithi05583006-26 from 420175 to 1050459 tid 1 2023-11-28T14:32:11.570 INFO:tasks.rados.rados.0.smithi055.stdout:26: writing smithi05583006-26 from 1796638 to 2267971 tid 2 2023-11-28T14:32:11.570 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.570 INFO:tasks.rados.rados.0.smithi055.stdout:19: finishing write tid 1 to smithi05583006-19 2023-11-28T14:32:11.570 INFO:tasks.rados.rados.0.smithi055.stdout:10: finishing write tid 3 to smithi05583006-10 2023-11-28T14:32:11.570 INFO:tasks.rados.rados.0.smithi055.stdout:10: finishing write tid 4 to smithi05583006-10 2023-11-28T14:32:11.570 INFO:tasks.rados.rados.0.smithi055.stdout:10: oid 10 updating version 0 to 3 2023-11-28T14:32:11.570 INFO:tasks.rados.rados.0.smithi055.stdout:10: oid 10 version 3 is already newer than 1 2023-11-28T14:32:11.570 INFO:tasks.rados.rados.0.smithi055.stdout:10: oid 10 version 3 is already newer than 2 2023-11-28T14:32:11.570 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2023-11-28T14:32:11.570 INFO:tasks.rados.rados.0.smithi055.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2023-11-28T14:32:11.571 INFO:tasks.rados.rados.0.smithi055.stdout:23: finishing write tid 1 to smithi05583006-23 2023-11-28T14:32:11.571 INFO:tasks.rados.rados.0.smithi055.stdout:20: finishing write tid 3 to smithi05583006-20 2023-11-28T14:32:11.571 INFO:tasks.rados.rados.0.smithi055.stdout:10: done (15 left) 2023-11-28T14:32:11.571 INFO:tasks.rados.rados.0.smithi055.stdout:27: write initial oid 27 2023-11-28T14:32:11.571 INFO:tasks.rados.rados.0.smithi055.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2023-11-28T14:32:11.576 INFO:tasks.rados.rados.0.smithi055.stdout:27: writing smithi05583006-27 from 436982 to 1142515 tid 1 2023-11-28T14:32:11.584 INFO:tasks.rados.rados.0.smithi055.stdout:27: writing smithi05583006-27 from 1855120 to 2436982 tid 2 2023-11-28T14:32:11.584 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.584 INFO:tasks.rados.rados.0.smithi055.stdout:20: finishing write tid 4 to smithi05583006-20 2023-11-28T14:32:11.584 INFO:tasks.rados.rados.0.smithi055.stdout:20: oid 20 updating version 0 to 2 2023-11-28T14:32:11.584 INFO:tasks.rados.rados.0.smithi055.stdout:20: oid 20 updating version 2 to 3 2023-11-28T14:32:11.584 INFO:tasks.rados.rados.0.smithi055.stdout:20: oid 20 version 3 is already newer than 1 2023-11-28T14:32:11.584 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2023-11-28T14:32:11.584 INFO:tasks.rados.rados.0.smithi055.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2023-11-28T14:32:11.584 INFO:tasks.rados.rados.0.smithi055.stdout:20: done (15 left) 2023-11-28T14:32:11.585 INFO:tasks.rados.rados.0.smithi055.stdout:28: write initial oid 28 2023-11-28T14:32:11.585 INFO:tasks.rados.rados.0.smithi055.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2023-11-28T14:32:11.590 INFO:tasks.rados.rados.0.smithi055.stdout:28: writing smithi05583006-28 from 453789 to 1234571 tid 1 2023-11-28T14:32:11.597 INFO:tasks.rados.rados.0.smithi055.stdout:28: writing smithi05583006-28 from 1913602 to 2388604 tid 2 2023-11-28T14:32:11.597 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.597 INFO:tasks.rados.rados.0.smithi055.stdout:14: finishing write tid 1 to smithi05583006-14 2023-11-28T14:32:11.597 INFO:tasks.rados.rados.0.smithi055.stdout:9: finishing write tid 3 to smithi05583006-9 2023-11-28T14:32:11.598 INFO:tasks.rados.rados.0.smithi055.stdout:9: finishing write tid 4 to smithi05583006-9 2023-11-28T14:32:11.598 INFO:tasks.rados.rados.0.smithi055.stdout:9: oid 9 updating version 0 to 2 2023-11-28T14:32:11.598 INFO:tasks.rados.rados.0.smithi055.stdout:9: oid 9 version 2 is already newer than 1 2023-11-28T14:32:11.598 INFO:tasks.rados.rados.0.smithi055.stdout:9: oid 9 updating version 2 to 3 2023-11-28T14:32:11.598 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2023-11-28T14:32:11.598 INFO:tasks.rados.rados.0.smithi055.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2023-11-28T14:32:11.598 INFO:tasks.rados.rados.0.smithi055.stdout:17: finishing write tid 3 to smithi05583006-17 2023-11-28T14:32:11.598 INFO:tasks.rados.rados.0.smithi055.stdout:17: finishing write tid 4 to smithi05583006-17 2023-11-28T14:32:11.598 INFO:tasks.rados.rados.0.smithi055.stdout:17: oid 17 updating version 0 to 3 2023-11-28T14:32:11.598 INFO:tasks.rados.rados.0.smithi055.stdout:17: oid 17 version 3 is already newer than 1 2023-11-28T14:32:11.598 INFO:tasks.rados.rados.0.smithi055.stdout:17: oid 17 version 3 is already newer than 2 2023-11-28T14:32:11.599 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2023-11-28T14:32:11.599 INFO:tasks.rados.rados.0.smithi055.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2023-11-28T14:32:11.599 INFO:tasks.rados.rados.0.smithi055.stdout:9: done (15 left) 2023-11-28T14:32:11.599 INFO:tasks.rados.rados.0.smithi055.stdout:17: done (14 left) 2023-11-28T14:32:11.599 INFO:tasks.rados.rados.0.smithi055.stdout:29: write initial oid 29 2023-11-28T14:32:11.599 INFO:tasks.rados.rados.0.smithi055.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2023-11-28T14:32:11.602 INFO:tasks.rados.rados.0.smithi055.stdout:29: writing smithi05583006-29 from 470596 to 926627 tid 1 2023-11-28T14:32:11.609 INFO:tasks.rados.rados.0.smithi055.stdout:29: writing smithi05583006-29 from 1572084 to 2190744 tid 2 2023-11-28T14:32:11.610 INFO:tasks.rados.rados.0.smithi055.stdout:30: write initial oid 30 2023-11-28T14:32:11.610 INFO:tasks.rados.rados.0.smithi055.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2023-11-28T14:32:11.616 INFO:tasks.rados.rados.0.smithi055.stdout:30: writing smithi05583006-30 from 487403 to 1018683 tid 1 2023-11-28T14:32:11.622 INFO:tasks.rados.rados.0.smithi055.stdout:30: writing smithi05583006-30 from 1514213 to 1992884 tid 2 2023-11-28T14:32:11.622 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.622 INFO:tasks.rados.rados.0.smithi055.stdout:24: finishing write tid 2 to smithi05583006-24 2023-11-28T14:32:11.622 INFO:tasks.rados.rados.0.smithi055.stdout:24: finishing write tid 3 to smithi05583006-24 2023-11-28T14:32:11.622 INFO:tasks.rados.rados.0.smithi055.stdout:21: finishing write tid 2 to smithi05583006-21 2023-11-28T14:32:11.622 INFO:tasks.rados.rados.0.smithi055.stdout:24: finishing write tid 4 to smithi05583006-24 2023-11-28T14:32:11.622 INFO:tasks.rados.rados.0.smithi055.stdout:24: oid 24 updating version 0 to 6 2023-11-28T14:32:11.622 INFO:tasks.rados.rados.0.smithi055.stdout:24: oid 24 version 6 is already newer than 6 2023-11-28T14:32:11.622 INFO:tasks.rados.rados.0.smithi055.stdout:24: oid 24 version 6 is already newer than 4 2023-11-28T14:32:11.622 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 24 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2023-11-28T14:32:11.623 INFO:tasks.rados.rados.0.smithi055.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2023-11-28T14:32:11.623 INFO:tasks.rados.rados.0.smithi055.stdout:18: finishing write tid 1 to smithi05583006-18 2023-11-28T14:32:11.623 INFO:tasks.rados.rados.0.smithi055.stdout:24: done (15 left) 2023-11-28T14:32:11.623 INFO:tasks.rados.rados.0.smithi055.stdout:31: write initial oid 31 2023-11-28T14:32:11.623 INFO:tasks.rados.rados.0.smithi055.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2023-11-28T14:32:11.625 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:11.622+0000 7f63d3b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:11.629 INFO:tasks.rados.rados.0.smithi055.stdout:31: writing smithi05583006-31 from 504210 to 1110739 tid 1 2023-11-28T14:32:11.631 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:11.630+0000 7f63d3b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:11.636 INFO:tasks.rados.rados.0.smithi055.stdout:31: writing smithi05583006-31 from 1572695 to 2195024 tid 2 2023-11-28T14:32:11.636 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.636 INFO:tasks.rados.rados.0.smithi055.stdout:11: finishing write tid 1 to smithi05583006-11 2023-11-28T14:32:11.636 INFO:tasks.rados.rados.0.smithi055.stdout:19: finishing write tid 2 to smithi05583006-19 2023-11-28T14:32:11.636 INFO:tasks.rados.rados.0.smithi055.stdout:27: finishing write tid 1 to smithi05583006-27 2023-11-28T14:32:11.638 INFO:tasks.rados.rados.0.smithi055.stdout:14: finishing write tid 2 to smithi05583006-14 2023-11-28T14:32:11.641 INFO:tasks.rados.rados.0.smithi055.stdout:14: finishing write tid 3 to smithi05583006-14 2023-11-28T14:32:11.648 INFO:tasks.rados.rados.0.smithi055.stdout:23: finishing write tid 2 to smithi05583006-23 2023-11-28T14:32:11.653 INFO:tasks.rados.rados.0.smithi055.stdout:14: finishing write tid 4 to smithi05583006-14 2023-11-28T14:32:11.653 INFO:tasks.rados.rados.0.smithi055.stdout:14: oid 14 updating version 0 to 7 2023-11-28T14:32:11.653 INFO:tasks.rados.rados.0.smithi055.stdout:14: oid 14 updating version 7 to 9 2023-11-28T14:32:11.653 INFO:tasks.rados.rados.0.smithi055.stdout:14: oid 14 version 9 is already newer than 9 2023-11-28T14:32:11.653 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 14 v 9 (ObjNum 13 snap 0 seq_num 13) dirty exists 2023-11-28T14:32:11.653 INFO:tasks.rados.rados.0.smithi055.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2023-11-28T14:32:11.653 INFO:tasks.rados.rados.0.smithi055.stdout:14: done (15 left) 2023-11-28T14:32:11.654 INFO:tasks.rados.rados.0.smithi055.stdout:32: write initial oid 32 2023-11-28T14:32:11.654 INFO:tasks.rados.rados.0.smithi055.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2023-11-28T14:32:11.660 INFO:tasks.rados.rados.0.smithi055.stdout:32: writing smithi05583006-32 from 521017 to 1319148 tid 1 2023-11-28T14:32:11.666 INFO:tasks.rados.rados.0.smithi055.stdout:32: writing smithi05583006-32 from 1747530 to 2229870 tid 2 2023-11-28T14:32:11.666 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.670 INFO:tasks.rados.rados.0.smithi055.stdout:21: finishing write tid 3 to smithi05583006-21 2023-11-28T14:32:11.678 INFO:tasks.rados.rados.0.smithi055.stdout:19: finishing write tid 3 to smithi05583006-19 2023-11-28T14:32:11.685 INFO:tasks.rados.rados.0.smithi055.stdout:21: finishing write tid 4 to smithi05583006-21 2023-11-28T14:32:11.685 INFO:tasks.rados.rados.0.smithi055.stdout:21: oid 21 updating version 0 to 6 2023-11-28T14:32:11.685 INFO:tasks.rados.rados.0.smithi055.stdout:21: oid 21 version 6 is already newer than 4 2023-11-28T14:32:11.685 INFO:tasks.rados.rados.0.smithi055.stdout:21: oid 21 version 6 is already newer than 5 2023-11-28T14:32:11.685 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2023-11-28T14:32:11.686 INFO:tasks.rados.rados.0.smithi055.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2023-11-28T14:32:11.686 INFO:tasks.rados.rados.0.smithi055.stdout:21: done (15 left) 2023-11-28T14:32:11.686 INFO:tasks.rados.rados.0.smithi055.stdout:33: write initial oid 33 2023-11-28T14:32:11.686 INFO:tasks.rados.rados.0.smithi055.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2023-11-28T14:32:11.690 INFO:tasks.rados.rados.0.smithi055.stdout:33: writing smithi05583006-33 from 537824 to 1011204 tid 1 2023-11-28T14:32:11.699 INFO:tasks.rados.rados.0.smithi055.stdout:33: writing smithi05583006-33 from 1806012 to 2432010 tid 2 2023-11-28T14:32:11.699 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.699 INFO:tasks.rados.rados.0.smithi055.stdout:19: finishing write tid 4 to smithi05583006-19 2023-11-28T14:32:11.700 INFO:tasks.rados.rados.0.smithi055.stdout:19: oid 19 updating version 0 to 2 2023-11-28T14:32:11.700 INFO:tasks.rados.rados.0.smithi055.stdout:19: oid 19 updating version 2 to 3 2023-11-28T14:32:11.700 INFO:tasks.rados.rados.0.smithi055.stdout:19: oid 19 version 3 is already newer than 1 2023-11-28T14:32:11.700 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2023-11-28T14:32:11.700 INFO:tasks.rados.rados.0.smithi055.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2023-11-28T14:32:11.700 INFO:tasks.rados.rados.0.smithi055.stdout:19: done (15 left) 2023-11-28T14:32:11.700 INFO:tasks.rados.rados.0.smithi055.stdout:34: write initial oid 34 2023-11-28T14:32:11.700 INFO:tasks.rados.rados.0.smithi055.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2023-11-28T14:32:11.705 INFO:tasks.rados.rados.0.smithi055.stdout:34: writing smithi05583006-34 from 554631 to 1103260 tid 1 2023-11-28T14:32:11.713 INFO:tasks.rados.rados.0.smithi055.stdout:34: writing smithi05583006-34 from 1748141 to 2234150 tid 2 2023-11-28T14:32:11.713 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.713 INFO:tasks.rados.rados.0.smithi055.stdout:27: finishing write tid 2 to smithi05583006-27 2023-11-28T14:32:11.713 INFO:tasks.rados.rados.0.smithi055.stdout:27: finishing write tid 3 to smithi05583006-27 2023-11-28T14:32:11.714 INFO:tasks.rados.rados.0.smithi055.stdout:27: finishing write tid 4 to smithi05583006-27 2023-11-28T14:32:11.714 INFO:tasks.rados.rados.0.smithi055.stdout:27: oid 27 updating version 0 to 6 2023-11-28T14:32:11.714 INFO:tasks.rados.rados.0.smithi055.stdout:27: oid 27 version 6 is already newer than 4 2023-11-28T14:32:11.714 INFO:tasks.rados.rados.0.smithi055.stdout:27: oid 27 version 6 is already newer than 6 2023-11-28T14:32:11.714 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2023-11-28T14:32:11.714 INFO:tasks.rados.rados.0.smithi055.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2023-11-28T14:32:11.714 INFO:tasks.rados.rados.0.smithi055.stdout:26: finishing write tid 1 to smithi05583006-26 2023-11-28T14:32:11.714 INFO:tasks.rados.rados.0.smithi055.stdout:27: done (15 left) 2023-11-28T14:32:11.714 INFO:tasks.rados.rados.0.smithi055.stdout:35: write initial oid 35 2023-11-28T14:32:11.714 INFO:tasks.rados.rados.0.smithi055.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2023-11-28T14:32:11.721 INFO:tasks.rados.rados.0.smithi055.stdout:35: writing smithi05583006-35 from 571438 to 1195316 tid 1 2023-11-28T14:32:11.730 INFO:tasks.rados.rados.0.smithi055.stdout:35: writing smithi05583006-35 from 1806623 to 2436290 tid 2 2023-11-28T14:32:11.730 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.730 INFO:tasks.rados.rados.0.smithi055.stdout:28: finishing write tid 1 to smithi05583006-28 2023-11-28T14:32:11.730 INFO:tasks.rados.rados.0.smithi055.stdout:23: finishing write tid 3 to smithi05583006-23 2023-11-28T14:32:11.730 INFO:tasks.rados.rados.0.smithi055.stdout:18: finishing write tid 2 to smithi05583006-18 2023-11-28T14:32:11.738 INFO:tasks.rados.rados.0.smithi055.stdout:23: finishing write tid 4 to smithi05583006-23 2023-11-28T14:32:11.738 INFO:tasks.rados.rados.0.smithi055.stdout:23: oid 23 updating version 0 to 6 2023-11-28T14:32:11.738 INFO:tasks.rados.rados.0.smithi055.stdout:23: oid 23 version 6 is already newer than 5 2023-11-28T14:32:11.738 INFO:tasks.rados.rados.0.smithi055.stdout:23: oid 23 version 6 is already newer than 4 2023-11-28T14:32:11.738 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2023-11-28T14:32:11.739 INFO:tasks.rados.rados.0.smithi055.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2023-11-28T14:32:11.739 INFO:tasks.rados.rados.0.smithi055.stdout:11: finishing write tid 2 to smithi05583006-11 2023-11-28T14:32:11.739 INFO:tasks.rados.rados.0.smithi055.stdout:23: done (15 left) 2023-11-28T14:32:11.739 INFO:tasks.rados.rados.0.smithi055.stdout:36: write initial oid 36 2023-11-28T14:32:11.739 INFO:tasks.rados.rados.0.smithi055.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2023-11-28T14:32:11.747 INFO:tasks.rados.rados.0.smithi055.stdout:36: writing smithi05583006-36 from 588245 to 1287372 tid 1 2023-11-28T14:32:11.753 INFO:tasks.rados.rados.0.smithi055.stdout:36: writing smithi05583006-36 from 1865105 to 2354783 tid 2 2023-11-28T14:32:11.753 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.753 INFO:tasks.rados.rados.0.smithi055.stdout:22: finishing write tid 1 to smithi05583006-22 2023-11-28T14:32:11.753 INFO:tasks.rados.rados.0.smithi055.stdout:11: finishing write tid 3 to smithi05583006-11 2023-11-28T14:32:11.758 INFO:tasks.rados.rados.0.smithi055.stdout:11: finishing write tid 4 to smithi05583006-11 2023-11-28T14:32:11.759 INFO:tasks.rados.rados.0.smithi055.stdout:11: oid 11 updating version 0 to 9 2023-11-28T14:32:11.759 INFO:tasks.rados.rados.0.smithi055.stdout:11: oid 11 version 9 is already newer than 7 2023-11-28T14:32:11.759 INFO:tasks.rados.rados.0.smithi055.stdout:11: oid 11 version 9 is already newer than 9 2023-11-28T14:32:11.759 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 11 v 9 (ObjNum 10 snap 0 seq_num 10) dirty exists 2023-11-28T14:32:11.759 INFO:tasks.rados.rados.0.smithi055.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2023-11-28T14:32:11.759 INFO:tasks.rados.rados.0.smithi055.stdout:11: done (15 left) 2023-11-28T14:32:11.759 INFO:tasks.rados.rados.0.smithi055.stdout:37: write initial oid 37 2023-11-28T14:32:11.759 INFO:tasks.rados.rados.0.smithi055.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2023-11-28T14:32:11.766 INFO:tasks.rados.rados.0.smithi055.stdout:37: writing smithi05583006-37 from 605052 to 1379428 tid 1 2023-11-28T14:32:11.773 INFO:tasks.rados.rados.0.smithi055.stdout:37: writing smithi05583006-37 from 1923587 to 2556923 tid 2 2023-11-28T14:32:11.773 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.775 INFO:tasks.rados.rados.0.smithi055.stdout:34: finishing write tid 1 to smithi05583006-34 2023-11-28T14:32:11.779 INFO:tasks.rados.rados.0.smithi055.stdout:30: finishing write tid 1 to smithi05583006-30 2023-11-28T14:32:11.789 INFO:tasks.rados.rados.0.smithi055.stdout:35: finishing write tid 1 to smithi05583006-35 2023-11-28T14:32:11.808 INFO:tasks.rados.rados.0.smithi055.stdout:28: finishing write tid 2 to smithi05583006-28 2023-11-28T14:32:11.832 INFO:tasks.rados.rados.0.smithi055.stdout:26: finishing write tid 2 to smithi05583006-26 2023-11-28T14:32:11.840 INFO:tasks.rados.rados.0.smithi055.stdout:13: finishing write tid 1 to smithi05583006-13 2023-11-28T14:32:11.840 INFO:tasks.rados.rados.0.smithi055.stdout:34: finishing write tid 2 to smithi05583006-34 2023-11-28T14:32:11.848 INFO:tasks.rados.rados.0.smithi055.stdout:18: finishing write tid 3 to smithi05583006-18 2023-11-28T14:32:11.857 INFO:tasks.rados.rados.0.smithi055.stdout:30: finishing write tid 2 to smithi05583006-30 2023-11-28T14:32:11.858 INFO:tasks.rados.rados.0.smithi055.stdout:37: finishing write tid 1 to smithi05583006-37 2023-11-28T14:32:11.859 INFO:tasks.rados.rados.0.smithi055.stdout:30: finishing write tid 3 to smithi05583006-30 2023-11-28T14:32:11.863 INFO:tasks.rados.rados.0.smithi055.stdout:18: finishing write tid 4 to smithi05583006-18 2023-11-28T14:32:11.863 INFO:tasks.rados.rados.0.smithi055.stdout:18: oid 18 updating version 0 to 5 2023-11-28T14:32:11.863 INFO:tasks.rados.rados.0.smithi055.stdout:18: oid 18 updating version 5 to 6 2023-11-28T14:32:11.863 INFO:tasks.rados.rados.0.smithi055.stdout:18: oid 18 version 6 is already newer than 4 2023-11-28T14:32:11.863 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-28T14:32:11.863 INFO:tasks.rados.rados.0.smithi055.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2023-11-28T14:32:11.864 INFO:tasks.rados.rados.0.smithi055.stdout:18: done (15 left) 2023-11-28T14:32:11.864 INFO:tasks.rados.rados.0.smithi055.stdout:38: write initial oid 38 2023-11-28T14:32:11.864 INFO:tasks.rados.rados.0.smithi055.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2023-11-28T14:32:11.870 INFO:tasks.rados.rados.0.smithi055.stdout:38: writing smithi05583006-38 from 621859 to 1071484 tid 1 2023-11-28T14:32:11.881 INFO:tasks.rados.rados.0.smithi055.stdout:38: writing smithi05583006-38 from 1865716 to 2621859 tid 2 2023-11-28T14:32:11.881 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.881 INFO:tasks.rados.rados.0.smithi055.stdout:22: finishing write tid 2 to smithi05583006-22 2023-11-28T14:32:11.881 INFO:tasks.rados.rados.0.smithi055.stdout:36: finishing write tid 1 to smithi05583006-36 2023-11-28T14:32:11.881 INFO:tasks.rados.rados.0.smithi055.stdout:30: finishing write tid 4 to smithi05583006-30 2023-11-28T14:32:11.881 INFO:tasks.rados.rados.0.smithi055.stdout:30: oid 30 updating version 0 to 7 2023-11-28T14:32:11.882 INFO:tasks.rados.rados.0.smithi055.stdout:30: oid 30 updating version 7 to 9 2023-11-28T14:32:11.882 INFO:tasks.rados.rados.0.smithi055.stdout:30: oid 30 version 9 is already newer than 9 2023-11-28T14:32:11.882 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 30 v 9 (ObjNum 29 snap 0 seq_num 29) dirty exists 2023-11-28T14:32:11.882 INFO:tasks.rados.rados.0.smithi055.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2023-11-28T14:32:11.882 INFO:tasks.rados.rados.0.smithi055.stdout:35: finishing write tid 2 to smithi05583006-35 2023-11-28T14:32:11.882 INFO:tasks.rados.rados.0.smithi055.stdout:30: done (15 left) 2023-11-28T14:32:11.882 INFO:tasks.rados.rados.0.smithi055.stdout:39: write initial oid 39 2023-11-28T14:32:11.882 INFO:tasks.rados.rados.0.smithi055.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2023-11-28T14:32:11.888 INFO:tasks.rados.rados.0.smithi055.stdout:39: writing smithi05583006-39 from 638666 to 1163540 tid 1 2023-11-28T14:32:11.897 INFO:tasks.rados.rados.0.smithi055.stdout:39: writing smithi05583006-39 from 1924198 to 2561203 tid 2 2023-11-28T14:32:11.897 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.900 INFO:tasks.rados.rados.0.smithi055.stdout:28: finishing write tid 3 to smithi05583006-28 2023-11-28T14:32:11.902 INFO:tasks.rados.rados.0.smithi055.stdout:34: finishing write tid 3 to smithi05583006-34 2023-11-28T14:32:11.905 INFO:tasks.rados.rados.0.smithi055.stdout:34: finishing write tid 4 to smithi05583006-34 2023-11-28T14:32:11.905 INFO:tasks.rados.rados.0.smithi055.stdout:34: oid 34 updating version 0 to 2 2023-11-28T14:32:11.906 INFO:tasks.rados.rados.0.smithi055.stdout:34: oid 34 updating version 2 to 3 2023-11-28T14:32:11.906 INFO:tasks.rados.rados.0.smithi055.stdout:34: oid 34 version 3 is already newer than 1 2023-11-28T14:32:11.906 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 34 v 3 (ObjNum 33 snap 0 seq_num 33) dirty exists 2023-11-28T14:32:11.906 INFO:tasks.rados.rados.0.smithi055.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2023-11-28T14:32:11.906 INFO:tasks.rados.rados.0.smithi055.stdout:34: done (15 left) 2023-11-28T14:32:11.906 INFO:tasks.rados.rados.0.smithi055.stdout:40: write initial oid 40 2023-11-28T14:32:11.906 INFO:tasks.rados.rados.0.smithi055.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2023-11-28T14:32:11.913 INFO:tasks.rados.rados.0.smithi055.stdout:40: writing smithi05583006-40 from 655473 to 1371949 tid 1 2023-11-28T14:32:11.923 INFO:tasks.rados.rados.0.smithi055.stdout:40: writing smithi05583006-40 from 2099033 to 2655473 tid 2 2023-11-28T14:32:11.923 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.923 INFO:tasks.rados.rados.0.smithi055.stdout:28: finishing write tid 4 to smithi05583006-28 2023-11-28T14:32:11.923 INFO:tasks.rados.rados.0.smithi055.stdout:28: oid 28 updating version 0 to 6 2023-11-28T14:32:11.923 INFO:tasks.rados.rados.0.smithi055.stdout:28: oid 28 version 6 is already newer than 4 2023-11-28T14:32:11.923 INFO:tasks.rados.rados.0.smithi055.stdout:28: oid 28 version 6 is already newer than 5 2023-11-28T14:32:11.923 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2023-11-28T14:32:11.923 INFO:tasks.rados.rados.0.smithi055.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2023-11-28T14:32:11.923 INFO:tasks.rados.rados.0.smithi055.stdout:13: finishing write tid 2 to smithi05583006-13 2023-11-28T14:32:11.923 INFO:tasks.rados.rados.0.smithi055.stdout:38: finishing write tid 1 to smithi05583006-38 2023-11-28T14:32:11.924 INFO:tasks.rados.rados.0.smithi055.stdout:13: finishing write tid 3 to smithi05583006-13 2023-11-28T14:32:11.924 INFO:tasks.rados.rados.0.smithi055.stdout:28: done (15 left) 2023-11-28T14:32:11.924 INFO:tasks.rados.rados.0.smithi055.stdout:41: write initial oid 41 2023-11-28T14:32:11.924 INFO:tasks.rados.rados.0.smithi055.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2023-11-28T14:32:11.931 INFO:tasks.rados.rados.0.smithi055.stdout:41: writing smithi05583006-41 from 672280 to 1464005 tid 1 2023-11-28T14:32:11.939 INFO:tasks.rados.rados.0.smithi055.stdout:41: writing smithi05583006-41 from 2157515 to 2672280 tid 2 2023-11-28T14:32:11.939 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.939 INFO:tasks.rados.rados.0.smithi055.stdout:26: finishing write tid 3 to smithi05583006-26 2023-11-28T14:32:11.939 INFO:tasks.rados.rados.0.smithi055.stdout:26: finishing write tid 4 to smithi05583006-26 2023-11-28T14:32:11.939 INFO:tasks.rados.rados.0.smithi055.stdout:26: oid 26 updating version 0 to 1 2023-11-28T14:32:11.939 INFO:tasks.rados.rados.0.smithi055.stdout:26: oid 26 updating version 1 to 2 2023-11-28T14:32:11.940 INFO:tasks.rados.rados.0.smithi055.stdout:26: oid 26 updating version 2 to 3 2023-11-28T14:32:11.940 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 26 v 3 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-11-28T14:32:11.940 INFO:tasks.rados.rados.0.smithi055.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2023-11-28T14:32:11.940 INFO:tasks.rados.rados.0.smithi055.stdout:37: finishing write tid 2 to smithi05583006-37 2023-11-28T14:32:11.940 INFO:tasks.rados.rados.0.smithi055.stdout:13: finishing write tid 4 to smithi05583006-13 2023-11-28T14:32:11.940 INFO:tasks.rados.rados.0.smithi055.stdout:13: oid 13 updating version 0 to 12 2023-11-28T14:32:11.940 INFO:tasks.rados.rados.0.smithi055.stdout:13: oid 13 version 12 is already newer than 10 2023-11-28T14:32:11.940 INFO:tasks.rados.rados.0.smithi055.stdout:13: oid 13 version 12 is already newer than 12 2023-11-28T14:32:11.940 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 13 v 12 (ObjNum 12 snap 0 seq_num 12) dirty exists 2023-11-28T14:32:11.940 INFO:tasks.rados.rados.0.smithi055.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2023-11-28T14:32:11.940 INFO:tasks.rados.rados.0.smithi055.stdout:29: finishing write tid 1 to smithi05583006-29 2023-11-28T14:32:11.941 INFO:tasks.rados.rados.0.smithi055.stdout:13: done (15 left) 2023-11-28T14:32:11.941 INFO:tasks.rados.rados.0.smithi055.stdout:26: done (14 left) 2023-11-28T14:32:11.941 INFO:tasks.rados.rados.0.smithi055.stdout:42: write initial oid 42 2023-11-28T14:32:11.941 INFO:tasks.rados.rados.0.smithi055.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2023-11-28T14:32:11.945 INFO:tasks.rados.rados.0.smithi055.stdout:42: writing smithi05583006-42 from 689087 to 1156061 tid 1 2023-11-28T14:32:11.952 INFO:tasks.rados.rados.0.smithi055.stdout:42: writing smithi05583006-42 from 1699644 to 2483976 tid 2 2023-11-28T14:32:11.953 INFO:tasks.rados.rados.0.smithi055.stdout:43: write initial oid 43 2023-11-28T14:32:11.953 INFO:tasks.rados.rados.0.smithi055.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2023-11-28T14:32:11.959 INFO:tasks.rados.rados.0.smithi055.stdout:43: writing smithi05583006-43 from 705894 to 1248117 tid 1 2023-11-28T14:32:11.967 INFO:tasks.rados.rados.0.smithi055.stdout:43: writing smithi05583006-43 from 1758126 to 2402469 tid 2 2023-11-28T14:32:11.967 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.967 INFO:tasks.rados.rados.0.smithi055.stdout:22: finishing write tid 3 to smithi05583006-22 2023-11-28T14:32:11.967 INFO:tasks.rados.rados.0.smithi055.stdout:22: finishing write tid 4 to smithi05583006-22 2023-11-28T14:32:11.968 INFO:tasks.rados.rados.0.smithi055.stdout:22: oid 22 updating version 0 to 11 2023-11-28T14:32:11.968 INFO:tasks.rados.rados.0.smithi055.stdout:22: oid 22 updating version 11 to 12 2023-11-28T14:32:11.968 INFO:tasks.rados.rados.0.smithi055.stdout:22: oid 22 version 12 is already newer than 10 2023-11-28T14:32:11.968 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 22 v 12 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-11-28T14:32:11.968 INFO:tasks.rados.rados.0.smithi055.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2023-11-28T14:32:11.968 INFO:tasks.rados.rados.0.smithi055.stdout:35: finishing write tid 3 to smithi05583006-35 2023-11-28T14:32:11.968 INFO:tasks.rados.rados.0.smithi055.stdout:36: finishing write tid 2 to smithi05583006-36 2023-11-28T14:32:11.968 INFO:tasks.rados.rados.0.smithi055.stdout:35: finishing write tid 4 to smithi05583006-35 2023-11-28T14:32:11.968 INFO:tasks.rados.rados.0.smithi055.stdout:35: oid 35 updating version 0 to 5 2023-11-28T14:32:11.968 INFO:tasks.rados.rados.0.smithi055.stdout:35: oid 35 version 5 is already newer than 4 2023-11-28T14:32:11.968 INFO:tasks.rados.rados.0.smithi055.stdout:35: oid 35 updating version 5 to 6 2023-11-28T14:32:11.968 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2023-11-28T14:32:11.968 INFO:tasks.rados.rados.0.smithi055.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2023-11-28T14:32:11.969 INFO:tasks.rados.rados.0.smithi055.stdout:22: done (15 left) 2023-11-28T14:32:11.969 INFO:tasks.rados.rados.0.smithi055.stdout:35: done (14 left) 2023-11-28T14:32:11.969 INFO:tasks.rados.rados.0.smithi055.stdout:44: write initial oid 44 2023-11-28T14:32:11.969 INFO:tasks.rados.rados.0.smithi055.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2023-11-28T14:32:11.974 INFO:tasks.rados.rados.0.smithi055.stdout:44: writing smithi05583006-44 from 722701 to 1340173 tid 1 2023-11-28T14:32:11.981 INFO:tasks.rados.rados.0.smithi055.stdout:44: writing smithi05583006-44 from 1816608 to 2604609 tid 2 2023-11-28T14:32:11.982 INFO:tasks.rados.rados.0.smithi055.stdout:45: write initial oid 45 2023-11-28T14:32:11.982 INFO:tasks.rados.rados.0.smithi055.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2023-11-28T14:32:11.989 INFO:tasks.rados.rados.0.smithi055.stdout:45: writing smithi05583006-45 from 739508 to 1432229 tid 1 2023-11-28T14:32:11.996 INFO:tasks.rados.rados.0.smithi055.stdout:45: writing smithi05583006-45 from 1875090 to 2523102 tid 2 2023-11-28T14:32:11.996 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:11.996 INFO:tasks.rados.rados.0.smithi055.stdout:40: finishing write tid 1 to smithi05583006-40 2023-11-28T14:32:11.996 INFO:tasks.rados.rados.0.smithi055.stdout:37: finishing write tid 3 to smithi05583006-37 2023-11-28T14:32:11.996 INFO:tasks.rados.rados.0.smithi055.stdout:37: finishing write tid 4 to smithi05583006-37 2023-11-28T14:32:11.996 INFO:tasks.rados.rados.0.smithi055.stdout:37: oid 37 updating version 0 to 4 2023-11-28T14:32:11.996 INFO:tasks.rados.rados.0.smithi055.stdout:37: oid 37 updating version 4 to 5 2023-11-28T14:32:11.996 INFO:tasks.rados.rados.0.smithi055.stdout:37: oid 37 updating version 5 to 6 2023-11-28T14:32:11.996 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2023-11-28T14:32:11.996 INFO:tasks.rados.rados.0.smithi055.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2023-11-28T14:32:11.996 INFO:tasks.rados.rados.0.smithi055.stdout:38: finishing write tid 2 to smithi05583006-38 2023-11-28T14:32:11.997 INFO:tasks.rados.rados.0.smithi055.stdout:37: done (15 left) 2023-11-28T14:32:11.997 INFO:tasks.rados.rados.0.smithi055.stdout:46: write initial oid 46 2023-11-28T14:32:11.997 INFO:tasks.rados.rados.0.smithi055.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2023-11-28T14:32:12.004 INFO:tasks.rados.rados.0.smithi055.stdout:46: writing smithi05583006-46 from 756315 to 1524285 tid 1 2023-11-28T14:32:12.012 INFO:tasks.rados.rados.0.smithi055.stdout:46: writing smithi05583006-46 from 1933572 to 2725242 tid 2 2023-11-28T14:32:12.012 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:12.012 INFO:tasks.rados.rados.0.smithi055.stdout:31: finishing write tid 1 to smithi05583006-31 2023-11-28T14:32:12.012 INFO:tasks.rados.rados.0.smithi055.stdout:36: finishing write tid 3 to smithi05583006-36 2023-11-28T14:32:12.012 INFO:tasks.rados.rados.0.smithi055.stdout:38: finishing write tid 3 to smithi05583006-38 2023-11-28T14:32:12.012 INFO:tasks.rados.rados.0.smithi055.stdout:38: finishing write tid 4 to smithi05583006-38 2023-11-28T14:32:12.013 INFO:tasks.rados.rados.0.smithi055.stdout:38: oid 38 updating version 0 to 12 2023-11-28T14:32:12.013 INFO:tasks.rados.rados.0.smithi055.stdout:38: oid 38 version 12 is already newer than 12 2023-11-28T14:32:12.013 INFO:tasks.rados.rados.0.smithi055.stdout:38: oid 38 version 12 is already newer than 10 2023-11-28T14:32:12.013 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-11-28T14:32:12.013 INFO:tasks.rados.rados.0.smithi055.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2023-11-28T14:32:12.013 INFO:tasks.rados.rados.0.smithi055.stdout:38: done (15 left) 2023-11-28T14:32:12.013 INFO:tasks.rados.rados.0.smithi055.stdout:47: write initial oid 47 2023-11-28T14:32:12.013 INFO:tasks.rados.rados.0.smithi055.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2023-11-28T14:32:12.019 INFO:tasks.rados.rados.0.smithi055.stdout:47: writing smithi05583006-47 from 773122 to 1332694 tid 1 2023-11-28T14:32:12.028 INFO:tasks.rados.rados.0.smithi055.stdout:47: writing smithi05583006-47 from 1992054 to 2643735 tid 2 2023-11-28T14:32:12.028 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:12.028 INFO:tasks.rados.rados.0.smithi055.stdout:16: finishing write tid 1 to smithi05583006-16 2023-11-28T14:32:12.028 INFO:tasks.rados.rados.0.smithi055.stdout:40: finishing write tid 2 to smithi05583006-40 2023-11-28T14:32:12.029 INFO:tasks.rados.rados.0.smithi055.stdout:36: finishing write tid 4 to smithi05583006-36 2023-11-28T14:32:12.029 INFO:tasks.rados.rados.0.smithi055.stdout:36: oid 36 updating version 0 to 5 2023-11-28T14:32:12.029 INFO:tasks.rados.rados.0.smithi055.stdout:36: oid 36 version 5 is already newer than 4 2023-11-28T14:32:12.029 INFO:tasks.rados.rados.0.smithi055.stdout:36: oid 36 updating version 5 to 6 2023-11-28T14:32:12.029 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-11-28T14:32:12.029 INFO:tasks.rados.rados.0.smithi055.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2023-11-28T14:32:12.029 INFO:tasks.rados.rados.0.smithi055.stdout:36: done (15 left) 2023-11-28T14:32:12.029 INFO:tasks.rados.rados.0.smithi055.stdout:48: write initial oid 48 2023-11-28T14:32:12.029 INFO:tasks.rados.rados.0.smithi055.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2023-11-28T14:32:12.036 INFO:tasks.rados.rados.0.smithi055.stdout:48: writing smithi05583006-48 from 789929 to 1424750 tid 1 2023-11-28T14:32:12.045 INFO:tasks.rados.rados.0.smithi055.stdout:48: writing smithi05583006-48 from 2050536 to 2789929 tid 2 2023-11-28T14:32:12.045 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:12.045 INFO:tasks.rados.rados.0.smithi055.stdout:29: finishing write tid 2 to smithi05583006-29 2023-11-28T14:32:12.046 INFO:tasks.rados.rados.0.smithi055.stdout:29: finishing write tid 3 to smithi05583006-29 2023-11-28T14:32:12.046 INFO:tasks.rados.rados.0.smithi055.stdout:29: finishing write tid 4 to smithi05583006-29 2023-11-28T14:32:12.046 INFO:tasks.rados.rados.0.smithi055.stdout:29: oid 29 updating version 0 to 9 2023-11-28T14:32:12.046 INFO:tasks.rados.rados.0.smithi055.stdout:29: oid 29 version 9 is already newer than 9 2023-11-28T14:32:12.046 INFO:tasks.rados.rados.0.smithi055.stdout:29: oid 29 version 9 is already newer than 7 2023-11-28T14:32:12.046 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 29 v 9 (ObjNum 28 snap 0 seq_num 28) dirty exists 2023-11-28T14:32:12.046 INFO:tasks.rados.rados.0.smithi055.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2023-11-28T14:32:12.046 INFO:tasks.rados.rados.0.smithi055.stdout:39: finishing write tid 1 to smithi05583006-39 2023-11-28T14:32:12.046 INFO:tasks.rados.rados.0.smithi055.stdout:41: finishing write tid 1 to smithi05583006-41 2023-11-28T14:32:12.046 INFO:tasks.rados.rados.0.smithi055.stdout:29: done (15 left) 2023-11-28T14:32:12.046 INFO:tasks.rados.rados.0.smithi055.stdout:49: write initial oid 49 2023-11-28T14:32:12.046 INFO:tasks.rados.rados.0.smithi055.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2023-11-28T14:32:12.051 INFO:tasks.rados.rados.0.smithi055.stdout:49: writing smithi05583006-49 from 406736 to 1116806 tid 1 2023-11-28T14:32:12.060 INFO:tasks.rados.rados.0.smithi055.stdout:49: writing smithi05583006-49 from 1709018 to 2364368 tid 2 2023-11-28T14:32:12.060 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:12.060 INFO:tasks.rados.rados.0.smithi055.stdout:40: finishing write tid 3 to smithi05583006-40 2023-11-28T14:32:12.060 INFO:tasks.rados.rados.0.smithi055.stdout:40: finishing write tid 4 to smithi05583006-40 2023-11-28T14:32:12.060 INFO:tasks.rados.rados.0.smithi055.stdout:40: oid 40 updating version 0 to 3 2023-11-28T14:32:12.060 INFO:tasks.rados.rados.0.smithi055.stdout:40: oid 40 version 3 is already newer than 2 2023-11-28T14:32:12.060 INFO:tasks.rados.rados.0.smithi055.stdout:40: oid 40 version 3 is already newer than 1 2023-11-28T14:32:12.060 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 40 v 3 (ObjNum 39 snap 0 seq_num 39) dirty exists 2023-11-28T14:32:12.060 INFO:tasks.rados.rados.0.smithi055.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2023-11-28T14:32:12.060 INFO:tasks.rados.rados.0.smithi055.stdout:40: done (15 left) 2023-11-28T14:32:12.060 INFO:tasks.rados.rados.0.smithi055.stdout:50: write initial oid 50 2023-11-28T14:32:12.061 INFO:tasks.rados.rados.0.smithi055.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2023-11-28T14:32:12.066 INFO:tasks.rados.rados.0.smithi055.stdout:50: writing smithi05583006-50 from 423543 to 1208862 tid 1 2023-11-28T14:32:12.074 INFO:tasks.rados.rados.0.smithi055.stdout:50: writing smithi05583006-50 from 1767500 to 2566508 tid 2 2023-11-28T14:32:12.074 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:12.074 INFO:tasks.rados.rados.0.smithi055.stdout:45: finishing write tid 1 to smithi05583006-45 2023-11-28T14:32:12.074 INFO:tasks.rados.rados.0.smithi055.stdout:42: finishing write tid 1 to smithi05583006-42 2023-11-28T14:32:12.089 INFO:tasks.rados.rados.0.smithi055.stdout:47: finishing write tid 1 to smithi05583006-47 2023-11-28T14:32:12.096 INFO:tasks.rados.rados.0.smithi055.stdout:16: finishing write tid 2 to smithi05583006-16 2023-11-28T14:32:12.098 INFO:tasks.rados.rados.0.smithi055.stdout:16: finishing write tid 3 to smithi05583006-16 2023-11-28T14:32:12.102 INFO:tasks.rados.rados.0.smithi055.stdout:16: finishing write tid 4 to smithi05583006-16 2023-11-28T14:32:12.102 INFO:tasks.rados.rados.0.smithi055.stdout:16: oid 16 updating version 0 to 15 2023-11-28T14:32:12.102 INFO:tasks.rados.rados.0.smithi055.stdout:16: oid 16 version 15 is already newer than 15 2023-11-28T14:32:12.102 INFO:tasks.rados.rados.0.smithi055.stdout:16: oid 16 version 15 is already newer than 13 2023-11-28T14:32:12.102 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 16 v 15 (ObjNum 15 snap 0 seq_num 15) dirty exists 2023-11-28T14:32:12.102 INFO:tasks.rados.rados.0.smithi055.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2023-11-28T14:32:12.103 INFO:tasks.rados.rados.0.smithi055.stdout:16: done (15 left) 2023-11-28T14:32:12.103 INFO:tasks.rados.rados.0.smithi055.stdout:51: write (excl) oid 2 current snap is 0 2023-11-28T14:32:12.103 INFO:tasks.rados.rados.0.smithi055.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2023-11-28T14:32:12.107 INFO:tasks.rados.rados.0.smithi055.stdout:51: writing smithi05583006-2 from 440350 to 900918 tid 1 2023-11-28T14:32:12.112 INFO:tasks.rados.rados.0.smithi055.stdout:51: writing smithi05583006-2 from 1309629 to 1852295 tid 2 2023-11-28T14:32:12.119 INFO:tasks.rados.rados.0.smithi055.stdout:51: writing smithi05583006-2 from 2323973 to 2840350 tid 3 2023-11-28T14:32:12.119 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:12.119 INFO:tasks.rados.rados.0.smithi055.stdout:39: finishing write tid 2 to smithi05583006-39 2023-11-28T14:32:12.119 INFO:tasks.rados.rados.0.smithi055.stdout:31: finishing write tid 2 to smithi05583006-31 2023-11-28T14:32:12.126 INFO:tasks.rados.rados.0.smithi055.stdout:41: finishing write tid 2 to smithi05583006-41 2023-11-28T14:32:12.127 INFO:tasks.rados.rados.0.smithi055.stdout:45: finishing write tid 2 to smithi05583006-45 2023-11-28T14:32:12.147 INFO:tasks.rados.rados.0.smithi055.stdout:47: finishing write tid 2 to smithi05583006-47 2023-11-28T14:32:12.150 INFO:tasks.rados.rados.0.smithi055.stdout:47: finishing write tid 3 to smithi05583006-47 2023-11-28T14:32:12.150 INFO:tasks.rados.rados.0.smithi055.stdout:42: finishing write tid 2 to smithi05583006-42 2023-11-28T14:32:12.153 INFO:tasks.rados.rados.0.smithi055.stdout:47: finishing write tid 4 to smithi05583006-47 2023-11-28T14:32:12.153 INFO:tasks.rados.rados.0.smithi055.stdout:47: oid 47 updating version 0 to 7 2023-11-28T14:32:12.153 INFO:tasks.rados.rados.0.smithi055.stdout:47: oid 47 updating version 7 to 9 2023-11-28T14:32:12.153 INFO:tasks.rados.rados.0.smithi055.stdout:47: oid 47 version 9 is already newer than 9 2023-11-28T14:32:12.153 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 47 v 9 (ObjNum 46 snap 0 seq_num 46) dirty exists 2023-11-28T14:32:12.153 INFO:tasks.rados.rados.0.smithi055.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2023-11-28T14:32:12.153 INFO:tasks.rados.rados.0.smithi055.stdout:47: done (15 left) 2023-11-28T14:32:12.153 INFO:tasks.rados.rados.0.smithi055.stdout:52: write (excl) oid 14 current snap is 0 2023-11-28T14:32:12.153 INFO:tasks.rados.rados.0.smithi055.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2023-11-28T14:32:12.158 INFO:tasks.rados.rados.0.smithi055.stdout:52: writing smithi05583006-14 from 457157 to 992974 tid 1 2023-11-28T14:32:12.165 INFO:tasks.rados.rados.0.smithi055.stdout:52: writing smithi05583006-14 from 1768111 to 2170788 tid 2 2023-11-28T14:32:12.165 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:12.165 INFO:tasks.rados.rados.0.smithi055.stdout:25: finishing write tid 1 to smithi05583006-25 2023-11-28T14:32:12.166 INFO:tasks.rados.rados.0.smithi055.stdout:39: finishing write tid 3 to smithi05583006-39 2023-11-28T14:32:12.167 INFO:tasks.rados.rados.0.smithi055.stdout:39: finishing write tid 4 to smithi05583006-39 2023-11-28T14:32:12.167 INFO:tasks.rados.rados.0.smithi055.stdout:39: oid 39 updating version 0 to 8 2023-11-28T14:32:12.167 INFO:tasks.rados.rados.0.smithi055.stdout:39: oid 39 updating version 8 to 9 2023-11-28T14:32:12.167 INFO:tasks.rados.rados.0.smithi055.stdout:39: oid 39 version 9 is already newer than 7 2023-11-28T14:32:12.168 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2023-11-28T14:32:12.168 INFO:tasks.rados.rados.0.smithi055.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2023-11-28T14:32:12.168 INFO:tasks.rados.rados.0.smithi055.stdout:39: done (15 left) 2023-11-28T14:32:12.168 INFO:tasks.rados.rados.0.smithi055.stdout:53: read oid 20 snap -1 2023-11-28T14:32:12.168 INFO:tasks.rados.rados.0.smithi055.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2023-11-28T14:32:12.168 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:12.169 INFO:tasks.rados.rados.0.smithi055.stdout:31: finishing write tid 3 to smithi05583006-31 2023-11-28T14:32:12.172 INFO:tasks.rados.rados.0.smithi055.stdout:31: finishing write tid 4 to smithi05583006-31 2023-11-28T14:32:12.172 INFO:tasks.rados.rados.0.smithi055.stdout:31: oid 31 updating version 0 to 5 2023-11-28T14:32:12.172 INFO:tasks.rados.rados.0.smithi055.stdout:31: oid 31 version 5 is already newer than 4 2023-11-28T14:32:12.172 INFO:tasks.rados.rados.0.smithi055.stdout:31: oid 31 updating version 5 to 6 2023-11-28T14:32:12.173 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2023-11-28T14:32:12.173 INFO:tasks.rados.rados.0.smithi055.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2023-11-28T14:32:12.173 INFO:tasks.rados.rados.0.smithi055.stdout:31: done (15 left) 2023-11-28T14:32:12.173 INFO:tasks.rados.rados.0.smithi055.stdout:54: read oid 13 snap -1 2023-11-28T14:32:12.173 INFO:tasks.rados.rados.0.smithi055.stdout:54: expect (ObjNum 12 snap 0 seq_num 12) 2023-11-28T14:32:12.173 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:12.180 INFO:tasks.rados.rados.0.smithi055.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2023-11-28T14:32:12.277 INFO:tasks.rados.rados.0.smithi055.stdout:45: finishing write tid 3 to smithi05583006-45 2023-11-28T14:32:12.277 INFO:tasks.rados.rados.0.smithi055.stdout:45: finishing write tid 4 to smithi05583006-45 2023-11-28T14:32:12.277 INFO:tasks.rados.rados.0.smithi055.stdout:45: oid 45 updating version 0 to 7 2023-11-28T14:32:12.277 INFO:tasks.rados.rados.0.smithi055.stdout:45: oid 45 updating version 7 to 8 2023-11-28T14:32:12.277 INFO:tasks.rados.rados.0.smithi055.stdout:45: oid 45 updating version 8 to 9 2023-11-28T14:32:12.277 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2023-11-28T14:32:12.277 INFO:tasks.rados.rados.0.smithi055.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2023-11-28T14:32:12.277 INFO:tasks.rados.rados.0.smithi055.stdout:49: finishing write tid 1 to smithi05583006-49 2023-11-28T14:32:12.277 INFO:tasks.rados.rados.0.smithi055.stdout:41: finishing write tid 3 to smithi05583006-41 2023-11-28T14:32:12.277 INFO:tasks.rados.rados.0.smithi055.stdout:41: finishing write tid 4 to smithi05583006-41 2023-11-28T14:32:12.277 INFO:tasks.rados.rados.0.smithi055.stdout:41: oid 41 updating version 0 to 13 2023-11-28T14:32:12.278 INFO:tasks.rados.rados.0.smithi055.stdout:41: oid 41 updating version 13 to 14 2023-11-28T14:32:12.278 INFO:tasks.rados.rados.0.smithi055.stdout:41: oid 41 updating version 14 to 15 2023-11-28T14:32:12.278 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 41 v 15 (ObjNum 40 snap 0 seq_num 40) dirty exists 2023-11-28T14:32:12.278 INFO:tasks.rados.rados.0.smithi055.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2023-11-28T14:32:12.278 INFO:tasks.rados.rados.0.smithi055.stdout:42: finishing write tid 3 to smithi05583006-42 2023-11-28T14:32:12.278 INFO:tasks.rados.rados.0.smithi055.stdout:41: done (15 left) 2023-11-28T14:32:12.278 INFO:tasks.rados.rados.0.smithi055.stdout:45: done (14 left) 2023-11-28T14:32:12.278 INFO:tasks.rados.rados.0.smithi055.stdout:53: done (13 left) 2023-11-28T14:32:12.278 INFO:tasks.rados.rados.0.smithi055.stdout:55: read oid 41 snap -1 2023-11-28T14:32:12.278 INFO:tasks.rados.rados.0.smithi055.stdout:55: expect (ObjNum 40 snap 0 seq_num 40) 2023-11-28T14:32:12.278 INFO:tasks.rados.rados.0.smithi055.stdout:56: read oid 5 snap -1 2023-11-28T14:32:12.278 INFO:tasks.rados.rados.0.smithi055.stdout:56: expect (ObjNum 4 snap 0 seq_num 4) 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:57: delete oid 37 current snap is 0 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:25: finishing write tid 2 to smithi05583006-25 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:42: finishing write tid 4 to smithi05583006-42 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:42: oid 42 updating version 0 to 8 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:42: oid 42 updating version 8 to 9 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:42: oid 42 version 9 is already newer than 7 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:25: finishing write tid 3 to smithi05583006-25 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:25: finishing write tid 4 to smithi05583006-25 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:25: oid 25 updating version 0 to 18 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:25: oid 25 version 18 is already newer than 16 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:25: oid 25 version 18 is already newer than 18 2023-11-28T14:32:12.279 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 25 v 18 (ObjNum 24 snap 0 seq_num 24) dirty exists 2023-11-28T14:32:12.280 INFO:tasks.rados.rados.0.smithi055.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2023-11-28T14:32:12.280 INFO:tasks.rados.rados.0.smithi055.stdout:48: finishing write tid 1 to smithi05583006-48 2023-11-28T14:32:12.280 INFO:tasks.rados.rados.0.smithi055.stdout:49: finishing write tid 2 to smithi05583006-49 2023-11-28T14:32:12.280 INFO:tasks.rados.rados.0.smithi055.stdout:49: finishing write tid 3 to smithi05583006-49 2023-11-28T14:32:12.280 INFO:tasks.rados.rados.0.smithi055.stdout:49: finishing write tid 4 to smithi05583006-49 2023-11-28T14:32:12.280 INFO:tasks.rados.rados.0.smithi055.stdout:49: oid 49 updating version 0 to 6 2023-11-28T14:32:12.280 INFO:tasks.rados.rados.0.smithi055.stdout:49: oid 49 version 6 is already newer than 6 2023-11-28T14:32:12.280 INFO:tasks.rados.rados.0.smithi055.stdout:49: oid 49 version 6 is already newer than 4 2023-11-28T14:32:12.280 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 49 v 6 (ObjNum 48 snap 0 seq_num 48) dirty exists 2023-11-28T14:32:12.280 INFO:tasks.rados.rados.0.smithi055.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2023-11-28T14:32:12.280 INFO:tasks.rados.rados.0.smithi055.stdout:44: finishing write tid 1 to smithi05583006-44 2023-11-28T14:32:12.281 INFO:tasks.rados.rados.0.smithi055.stdout:43: finishing write tid 1 to smithi05583006-43 2023-11-28T14:32:12.282 INFO:tasks.rados.rados.0.smithi055.stdout:32: finishing write tid 1 to smithi05583006-32 2023-11-28T14:32:12.317 INFO:tasks.rados.rados.0.smithi055.stdout:32: finishing write tid 2 to smithi05583006-32 2023-11-28T14:32:12.318 INFO:tasks.rados.rados.0.smithi055.stdout:48: finishing write tid 2 to smithi05583006-48 2023-11-28T14:32:12.319 INFO:tasks.rados.rados.0.smithi055.stdout:48: finishing write tid 3 to smithi05583006-48 2023-11-28T14:32:12.319 INFO:tasks.rados.rados.0.smithi055.stdout:50: finishing write tid 1 to smithi05583006-50 2023-11-28T14:32:12.322 INFO:tasks.rados.rados.0.smithi055.stdout:32: finishing write tid 3 to smithi05583006-32 2023-11-28T14:32:12.322 INFO:tasks.rados.rados.0.smithi055.stdout:44: finishing write tid 2 to smithi05583006-44 2023-11-28T14:32:12.323 INFO:tasks.rados.rados.0.smithi055.stdout:48: finishing write tid 4 to smithi05583006-48 2023-11-28T14:32:12.323 INFO:tasks.rados.rados.0.smithi055.stdout:48: oid 48 updating version 0 to 10 2023-11-28T14:32:12.323 INFO:tasks.rados.rados.0.smithi055.stdout:48: oid 48 updating version 10 to 12 2023-11-28T14:32:12.324 INFO:tasks.rados.rados.0.smithi055.stdout:48: oid 48 version 12 is already newer than 12 2023-11-28T14:32:12.324 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-11-28T14:32:12.324 INFO:tasks.rados.rados.0.smithi055.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2023-11-28T14:32:12.324 INFO:tasks.rados.rados.0.smithi055.stdout:32: finishing write tid 4 to smithi05583006-32 2023-11-28T14:32:12.324 INFO:tasks.rados.rados.0.smithi055.stdout:32: oid 32 updating version 0 to 19 2023-11-28T14:32:12.324 INFO:tasks.rados.rados.0.smithi055.stdout:32: oid 32 updating version 19 to 21 2023-11-28T14:32:12.324 INFO:tasks.rados.rados.0.smithi055.stdout:32: oid 32 version 21 is already newer than 21 2023-11-28T14:32:12.324 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 32 v 21 (ObjNum 31 snap 0 seq_num 31) dirty exists 2023-11-28T14:32:12.324 INFO:tasks.rados.rados.0.smithi055.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2023-11-28T14:32:12.325 INFO:tasks.rados.rados.0.smithi055.stdout:44: finishing write tid 3 to smithi05583006-44 2023-11-28T14:32:12.340 INFO:tasks.rados.rados.0.smithi055.stdout:44: finishing write tid 4 to smithi05583006-44 2023-11-28T14:32:12.341 INFO:tasks.rados.rados.0.smithi055.stdout:44: oid 44 updating version 0 to 18 2023-11-28T14:32:12.341 INFO:tasks.rados.rados.0.smithi055.stdout:44: oid 44 version 18 is already newer than 16 2023-11-28T14:32:12.341 INFO:tasks.rados.rados.0.smithi055.stdout:44: oid 44 version 18 is already newer than 18 2023-11-28T14:32:12.341 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 44 v 18 (ObjNum 43 snap 0 seq_num 43) dirty exists 2023-11-28T14:32:12.341 INFO:tasks.rados.rados.0.smithi055.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2023-11-28T14:32:12.341 INFO:tasks.rados.rados.0.smithi055.stdout:43: finishing write tid 2 to smithi05583006-43 2023-11-28T14:32:12.351 INFO:tasks.rados.rados.0.smithi055.stdout:43: finishing write tid 3 to smithi05583006-43 2023-11-28T14:32:12.351 INFO:tasks.rados.rados.0.smithi055.stdout:43: finishing write tid 4 to smithi05583006-43 2023-11-28T14:32:12.351 INFO:tasks.rados.rados.0.smithi055.stdout:43: oid 43 updating version 0 to 12 2023-11-28T14:32:12.352 INFO:tasks.rados.rados.0.smithi055.stdout:43: oid 43 version 12 is already newer than 12 2023-11-28T14:32:12.352 INFO:tasks.rados.rados.0.smithi055.stdout:43: oid 43 version 12 is already newer than 10 2023-11-28T14:32:12.352 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 43 v 12 (ObjNum 42 snap 0 seq_num 42) dirty exists 2023-11-28T14:32:12.352 INFO:tasks.rados.rados.0.smithi055.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2023-11-28T14:32:12.361 INFO:tasks.rados.rados.0.smithi055.stdout:33: finishing write tid 1 to smithi05583006-33 2023-11-28T14:32:12.363 INFO:tasks.rados.rados.0.smithi055.stdout:50: finishing write tid 2 to smithi05583006-50 2023-11-28T14:32:12.368 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:32:12.368 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:32:12.369 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-11-28T14:32:12.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-11-28T14:32:12.372 INFO:tasks.rados.rados.0.smithi055.stdout:51: finishing write tid 1 to smithi05583006-2 2023-11-28T14:32:12.392 INFO:tasks.rados.rados.0.smithi055.stdout:25: done (15 left) 2023-11-28T14:32:12.392 INFO:tasks.rados.rados.0.smithi055.stdout:32: done (14 left) 2023-11-28T14:32:12.392 INFO:tasks.rados.rados.0.smithi055.stdout:42: done (13 left) 2023-11-28T14:32:12.392 INFO:tasks.rados.rados.0.smithi055.stdout:43: done (12 left) 2023-11-28T14:32:12.392 INFO:tasks.rados.rados.0.smithi055.stdout:44: done (11 left) 2023-11-28T14:32:12.393 INFO:tasks.rados.rados.0.smithi055.stdout:48: done (10 left) 2023-11-28T14:32:12.393 INFO:tasks.rados.rados.0.smithi055.stdout:49: done (9 left) 2023-11-28T14:32:12.393 INFO:tasks.rados.rados.0.smithi055.stdout:57: done (8 left) 2023-11-28T14:32:12.393 INFO:tasks.rados.rados.0.smithi055.stdout:58: read oid 39 snap -1 2023-11-28T14:32:12.393 INFO:tasks.rados.rados.0.smithi055.stdout:58: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-28T14:32:12.393 INFO:tasks.rados.rados.0.smithi055.stdout:59: write oid 35 current snap is 0 2023-11-28T14:32:12.393 INFO:tasks.rados.rados.0.smithi055.stdout:59: seq_num 53 ranges {490771=686315,1885075=406346} 2023-11-28T14:32:12.398 INFO:tasks.rados.rados.0.smithi055.stdout:59: writing smithi05583006-35 from 490771 to 1177086 tid 1 2023-11-28T14:32:12.406 INFO:tasks.rados.rados.0.smithi055.stdout:59: writing smithi05583006-35 from 1885075 to 2291421 tid 2 2023-11-28T14:32:12.406 INFO:tasks.rados.rados.0.smithi055.stdout:60: read oid 8 snap -1 2023-11-28T14:32:12.406 INFO:tasks.rados.rados.0.smithi055.stdout:60: expect (ObjNum 7 snap 0 seq_num 7) 2023-11-28T14:32:12.406 INFO:tasks.rados.rados.0.smithi055.stdout:61: read oid 28 snap -1 2023-11-28T14:32:12.406 INFO:tasks.rados.rados.0.smithi055.stdout:61: expect (ObjNum 27 snap 0 seq_num 27) 2023-11-28T14:32:12.406 INFO:tasks.rados.rados.0.smithi055.stdout:62: read oid 16 snap -1 2023-11-28T14:32:12.406 INFO:tasks.rados.rados.0.smithi055.stdout:62: expect (ObjNum 15 snap 0 seq_num 15) 2023-11-28T14:32:12.406 INFO:tasks.rados.rados.0.smithi055.stdout:63: read oid 36 snap -1 2023-11-28T14:32:12.406 INFO:tasks.rados.rados.0.smithi055.stdout:63: expect (ObjNum 35 snap 0 seq_num 35) 2023-11-28T14:32:12.406 INFO:tasks.rados.rados.0.smithi055.stdout:64: read oid 11 snap -1 2023-11-28T14:32:12.407 INFO:tasks.rados.rados.0.smithi055.stdout:64: expect (ObjNum 10 snap 0 seq_num 10) 2023-11-28T14:32:12.407 INFO:tasks.rados.rados.0.smithi055.stdout:65: snap_create 2023-11-28T14:32:12.413 INFO:tasks.rados.rados.0.smithi055.stdout:33: finishing write tid 2 to smithi05583006-33 2023-11-28T14:32:12.415 INFO:tasks.rados.rados.0.smithi055.stdout:58: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-28T14:32:12.527 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:12.520+0000 7fab5211e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:12.532 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:12.531+0000 7fab5211e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:12.557 INFO:tasks.rados.rados.0.smithi055.stdout:50: finishing write tid 3 to smithi05583006-50 2023-11-28T14:32:12.557 INFO:tasks.rados.rados.0.smithi055.stdout:50: finishing write tid 4 to smithi05583006-50 2023-11-28T14:32:12.557 INFO:tasks.rados.rados.0.smithi055.stdout:50: oid 50 updating version 0 to 7 2023-11-28T14:32:12.557 INFO:tasks.rados.rados.0.smithi055.stdout:50: oid 50 updating version 7 to 9 2023-11-28T14:32:12.557 INFO:tasks.rados.rados.0.smithi055.stdout:50: oid 50 version 9 is already newer than 8 2023-11-28T14:32:12.557 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2023-11-28T14:32:12.557 INFO:tasks.rados.rados.0.smithi055.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2023-11-28T14:32:12.557 INFO:tasks.rados.rados.0.smithi055.stdout:51: finishing write tid 2 to smithi05583006-2 2023-11-28T14:32:12.557 INFO:tasks.rados.rados.0.smithi055.stdout:33: finishing write tid 3 to smithi05583006-33 2023-11-28T14:32:12.558 INFO:tasks.rados.rados.0.smithi055.stdout:33: finishing write tid 4 to smithi05583006-33 2023-11-28T14:32:12.558 INFO:tasks.rados.rados.0.smithi055.stdout:33: oid 33 updating version 0 to 22 2023-11-28T14:32:12.558 INFO:tasks.rados.rados.0.smithi055.stdout:33: oid 33 updating version 22 to 24 2023-11-28T14:32:12.558 INFO:tasks.rados.rados.0.smithi055.stdout:33: oid 33 version 24 is already newer than 24 2023-11-28T14:32:12.558 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 33 v 24 (ObjNum 32 snap 0 seq_num 32) dirty exists 2023-11-28T14:32:12.558 INFO:tasks.rados.rados.0.smithi055.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2023-11-28T14:32:12.558 INFO:tasks.rados.rados.0.smithi055.stdout:51: finishing write tid 3 to smithi05583006-2 2023-11-28T14:32:12.558 INFO:tasks.rados.rados.0.smithi055.stdout:63: expect (ObjNum 35 snap 0 seq_num 35) 2023-11-28T14:32:12.571 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:12.571 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:12.571 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:12.575 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:12.575 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:12.575 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-11-28T14:32:12.656 INFO:tasks.rados.rados.0.smithi055.stdout:61: expect (ObjNum 27 snap 0 seq_num 27) 2023-11-28T14:32:12.728 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:12.723+0000 7f7cf766d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:12.731 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:12.730+0000 7f7cf766d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:12.744 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:12.744 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:12.744 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:12.744 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:12.744 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:12.744 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:12.744 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:12.744 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:12.744 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:12.744 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:12.745 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:12.745 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:12.746 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:12.747 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:12.747 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-11-28T14:32:12.751 INFO:tasks.rados.rados.0.smithi055.stdout:51: finishing write tid 4 to smithi05583006-2 2023-11-28T14:32:12.751 INFO:tasks.rados.rados.0.smithi055.stdout:51: finishing write tid 5 to smithi05583006-2 2023-11-28T14:32:12.751 INFO:tasks.rados.rados.0.smithi055.stdout:51: oid 2 updating version 0 to 22 2023-11-28T14:32:12.751 INFO:tasks.rados.rados.0.smithi055.stdout:51: oid 2 version 22 is already newer than 22 2023-11-28T14:32:12.751 INFO:tasks.rados.rados.0.smithi055.stdout:51: oid 2 version 22 is already newer than 22 2023-11-28T14:32:12.752 INFO:tasks.rados.rados.0.smithi055.stdout:51: oid 2 version 22 is already newer than 19 2023-11-28T14:32:12.752 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 2 v 22 (ObjNum 50 snap 0 seq_num 50) dirty exists 2023-11-28T14:32:12.752 INFO:tasks.rados.rados.0.smithi055.stdout:51: left oid 2 (ObjNum 50 snap 0 seq_num 50) 2023-11-28T14:32:12.752 INFO:tasks.rados.rados.0.smithi055.stdout:46: finishing write tid 1 to smithi05583006-46 2023-11-28T14:32:12.752 INFO:tasks.rados.rados.0.smithi055.stdout:59: finishing write tid 1 to smithi05583006-35 2023-11-28T14:32:12.752 INFO:tasks.rados.rados.0.smithi055.stdout:52: finishing write tid 1 to smithi05583006-14 2023-11-28T14:32:12.752 INFO:tasks.rados.rados.0.smithi055.stdout:46: finishing write tid 2 to smithi05583006-46 2023-11-28T14:32:12.752 INFO:tasks.rados.rados.0.smithi055.stdout:52: finishing write tid 2 to smithi05583006-14 2023-11-28T14:32:12.752 INFO:tasks.rados.rados.0.smithi055.stdout:46: finishing write tid 3 to smithi05583006-46 2023-11-28T14:32:12.752 INFO:tasks.rados.rados.0.smithi055.stdout:59: finishing write tid 2 to smithi05583006-35 2023-11-28T14:32:12.752 INFO:tasks.rados.rados.0.smithi055.stdout:46: finishing write tid 4 to smithi05583006-46 2023-11-28T14:32:12.753 INFO:tasks.rados.rados.0.smithi055.stdout:46: oid 46 updating version 0 to 27 2023-11-28T14:32:12.753 INFO:tasks.rados.rados.0.smithi055.stdout:46: oid 46 version 27 is already newer than 27 2023-11-28T14:32:12.753 INFO:tasks.rados.rados.0.smithi055.stdout:46: oid 46 version 27 is already newer than 25 2023-11-28T14:32:12.753 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 46 v 27 (ObjNum 45 snap 0 seq_num 45) dirty exists 2023-11-28T14:32:12.753 INFO:tasks.rados.rados.0.smithi055.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2023-11-28T14:32:12.753 INFO:tasks.rados.rados.0.smithi055.stdout:52: finishing write tid 3 to smithi05583006-14 2023-11-28T14:32:12.753 INFO:tasks.rados.rados.0.smithi055.stdout:54: expect (ObjNum 12 snap 0 seq_num 12) 2023-11-28T14:32:12.835 INFO:tasks.rados.rados.0.smithi055.stdout:52: finishing write tid 4 to smithi05583006-14 2023-11-28T14:32:12.835 INFO:tasks.rados.rados.0.smithi055.stdout:52: oid 14 updating version 0 to 23 2023-11-28T14:32:12.835 INFO:tasks.rados.rados.0.smithi055.stdout:52: oid 14 updating version 23 to 25 2023-11-28T14:32:12.835 INFO:tasks.rados.rados.0.smithi055.stdout:52: oid 14 version 25 is already newer than 25 2023-11-28T14:32:12.835 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 14 v 25 (ObjNum 51 snap 0 seq_num 51) dirty exists 2023-11-28T14:32:12.835 INFO:tasks.rados.rados.0.smithi055.stdout:52: left oid 14 (ObjNum 51 snap 0 seq_num 51) 2023-11-28T14:32:12.835 INFO:tasks.rados.rados.0.smithi055.stdout:56: expect (ObjNum 4 snap 0 seq_num 4) 2023-11-28T14:32:12.864 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:12.859+0000 7f70858ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:12.867 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:12.866+0000 7f70858ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:12.880 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:12.880 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:12.881 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:12.881 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:12.881 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:12.881 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:12.881 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:12.881 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:12.882 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:12.882 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:12.882 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:12.882 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:12.883 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:12.883 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:12.884 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-11-28T14:32:12.913 INFO:tasks.rados.rados.0.smithi055.stdout:59: finishing write tid 3 to smithi05583006-35 2023-11-28T14:32:12.914 INFO:tasks.rados.rados.0.smithi055.stdout:55: expect (ObjNum 40 snap 0 seq_num 40) 2023-11-28T14:32:12.996 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:12.991+0000 7f488875e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:13.000 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:12.999+0000 7f488875e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:13.013 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:13.013 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:13.013 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:13.013 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:13.013 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:13.014 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:13.014 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:13.014 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:13.014 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:13.014 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:13.014 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:13.014 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:13.016 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:13.018 INFO:tasks.rados.rados.0.smithi055.stdout:59: finishing write tid 4 to smithi05583006-35 2023-11-28T14:32:13.018 INFO:tasks.rados.rados.0.smithi055.stdout:59: oid 35 updating version 0 to 10 2023-11-28T14:32:13.018 INFO:tasks.rados.rados.0.smithi055.stdout:59: oid 35 updating version 10 to 12 2023-11-28T14:32:13.018 INFO:tasks.rados.rados.0.smithi055.stdout:59: oid 35 version 12 is already newer than 11 2023-11-28T14:32:13.018 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 35 v 12 (ObjNum 53 snap 0 seq_num 53) dirty exists 2023-11-28T14:32:13.018 INFO:tasks.rados.rados.0.smithi055.stdout:59: left oid 35 (ObjNum 53 snap 0 seq_num 53) 2023-11-28T14:32:13.018 INFO:tasks.rados.rados.0.smithi055.stdout:60: expect (ObjNum 7 snap 0 seq_num 7) 2023-11-28T14:32:13.100 INFO:tasks.rados.rados.0.smithi055.stdout:62: expect (ObjNum 15 snap 0 seq_num 15) 2023-11-28T14:32:13.187 INFO:tasks.rados.rados.0.smithi055.stdout:64: expect (ObjNum 10 snap 0 seq_num 10) 2023-11-28T14:32:13.271 INFO:tasks.rados.rados.0.smithi055.stdout:33: done (15 left) 2023-11-28T14:32:13.271 INFO:tasks.rados.rados.0.smithi055.stdout:46: done (14 left) 2023-11-28T14:32:13.271 INFO:tasks.rados.rados.0.smithi055.stdout:50: done (13 left) 2023-11-28T14:32:13.271 INFO:tasks.rados.rados.0.smithi055.stdout:51: done (12 left) 2023-11-28T14:32:13.271 INFO:tasks.rados.rados.0.smithi055.stdout:52: done (11 left) 2023-11-28T14:32:13.271 INFO:tasks.rados.rados.0.smithi055.stdout:54: done (10 left) 2023-11-28T14:32:13.271 INFO:tasks.rados.rados.0.smithi055.stdout:55: done (9 left) 2023-11-28T14:32:13.271 INFO:tasks.rados.rados.0.smithi055.stdout:56: done (8 left) 2023-11-28T14:32:13.271 INFO:tasks.rados.rados.0.smithi055.stdout:58: done (7 left) 2023-11-28T14:32:13.271 INFO:tasks.rados.rados.0.smithi055.stdout:59: done (6 left) 2023-11-28T14:32:13.271 INFO:tasks.rados.rados.0.smithi055.stdout:60: done (5 left) 2023-11-28T14:32:13.272 INFO:tasks.rados.rados.0.smithi055.stdout:61: done (4 left) 2023-11-28T14:32:13.272 INFO:tasks.rados.rados.0.smithi055.stdout:62: done (3 left) 2023-11-28T14:32:13.272 INFO:tasks.rados.rados.0.smithi055.stdout:63: done (2 left) 2023-11-28T14:32:13.272 INFO:tasks.rados.rados.0.smithi055.stdout:64: done (1 left) 2023-11-28T14:32:13.272 INFO:tasks.rados.rados.0.smithi055.stdout:65: done (0 left) 2023-11-28T14:32:13.272 INFO:tasks.rados.rados.0.smithi055.stdout:66: write oid 33 current snap is 1 2023-11-28T14:32:13.272 INFO:tasks.rados.rados.0.smithi055.stdout:66: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2023-11-28T14:32:13.275 INFO:tasks.rados.rados.0.smithi055.stdout:66: writing smithi05583006-33 from 507578 to 985495 tid 1 2023-11-28T14:32:13.282 INFO:tasks.rados.rados.0.smithi055.stdout:66: writing smithi05583006-33 from 1543557 to 2093561 tid 2 2023-11-28T14:32:13.287 INFO:tasks.rados.rados.0.smithi055.stdout:66: writing smithi05583006-33 from 2833665 to 2907578 tid 3 2023-11-28T14:32:13.287 INFO:tasks.rados.rados.0.smithi055.stdout:67: read oid 26 snap 0 2023-11-28T14:32:13.287 INFO:tasks.rados.rados.0.smithi055.stdout:67: expect (ObjNum 25 snap 0 seq_num 25) 2023-11-28T14:32:13.287 INFO:tasks.rados.rados.0.smithi055.stdout:68: read oid 6 snap 0 2023-11-28T14:32:13.287 INFO:tasks.rados.rados.0.smithi055.stdout:68: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-28T14:32:13.287 INFO:tasks.rados.rados.0.smithi055.stdout:69: write (excl) oid 47 current snap is 1 2023-11-28T14:32:13.287 INFO:tasks.rados.rados.0.smithi055.stdout:69: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2023-11-28T14:32:13.293 INFO:tasks.rados.rados.0.smithi055.stdout:69: writing smithi05583006-47 from 524385 to 1077551 tid 1 2023-11-28T14:32:13.299 INFO:tasks.rados.rados.0.smithi055.stdout:69: writing smithi05583006-47 from 1602039 to 2012054 tid 2 2023-11-28T14:32:13.306 INFO:tasks.rados.rados.0.smithi055.stdout:69: writing smithi05583006-47 from 2577441 to 2924385 tid 3 2023-11-28T14:32:13.306 INFO:tasks.rados.rados.0.smithi055.stdout:70: read oid 40 snap -1 2023-11-28T14:32:13.306 INFO:tasks.rados.rados.0.smithi055.stdout:70: expect (ObjNum 39 snap 0 seq_num 39) 2023-11-28T14:32:13.306 INFO:tasks.rados.rados.0.smithi055.stdout:71: read oid 36 snap 0 2023-11-28T14:32:13.307 INFO:tasks.rados.rados.0.smithi055.stdout:71: expect (ObjNum 35 snap 0 seq_num 35) 2023-11-28T14:32:13.307 INFO:tasks.rados.rados.0.smithi055.stdout:72: write oid 17 current snap is 1 2023-11-28T14:32:13.307 INFO:tasks.rados.rados.0.smithi055.stdout:72: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2023-11-28T14:32:13.314 INFO:tasks.rados.rados.0.smithi055.stdout:72: writing smithi05583006-17 from 541192 to 1169607 tid 1 2023-11-28T14:32:13.322 INFO:tasks.rados.rados.0.smithi055.stdout:72: writing smithi05583006-17 from 1660521 to 2214194 tid 2 2023-11-28T14:32:13.326 INFO:tasks.rados.rados.0.smithi055.stdout:72: writing smithi05583006-17 from 2888511 to 2941192 tid 3 2023-11-28T14:32:13.327 INFO:tasks.rados.rados.0.smithi055.stdout:73: write (excl) oid 49 current snap is 1 2023-11-28T14:32:13.327 INFO:tasks.rados.rados.0.smithi055.stdout:73: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2023-11-28T14:32:13.337 INFO:tasks.rados.rados.0.smithi055.stdout:73: writing smithi05583006-49 from 557999 to 1261663 tid 1 2023-11-28T14:32:13.341 INFO:tasks.rados.rados.0.smithi055.stdout:73: writing smithi05583006-49 from 1719003 to 2132687 tid 2 2023-11-28T14:32:13.347 INFO:tasks.rados.rados.0.smithi055.stdout:73: writing smithi05583006-49 from 2632287 to 2957999 tid 3 2023-11-28T14:32:13.347 INFO:tasks.rados.rados.0.smithi055.stdout:74: write oid 12 current snap is 1 2023-11-28T14:32:13.347 INFO:tasks.rados.rados.0.smithi055.stdout:74: seq_num 58 ranges {574806=778913,2061132=557342} 2023-11-28T14:32:13.355 INFO:tasks.rados.rados.0.smithi055.stdout:74: writing smithi05583006-12 from 574806 to 1353719 tid 1 2023-11-28T14:32:13.363 INFO:tasks.rados.rados.0.smithi055.stdout:74: writing smithi05583006-12 from 2061132 to 2618474 tid 2 2023-11-28T14:32:13.364 INFO:tasks.rados.rados.0.smithi055.stdout:75: read oid 9 snap -1 2023-11-28T14:32:13.364 INFO:tasks.rados.rados.0.smithi055.stdout:75: expect (ObjNum 8 snap 0 seq_num 8) 2023-11-28T14:32:13.364 INFO:tasks.rados.rados.0.smithi055.stdout:76: read oid 27 snap -1 2023-11-28T14:32:13.364 INFO:tasks.rados.rados.0.smithi055.stdout:76: expect (ObjNum 26 snap 0 seq_num 26) 2023-11-28T14:32:13.364 INFO:tasks.rados.rados.0.smithi055.stdout:77: write (excl) oid 5 current snap is 1 2023-11-28T14:32:13.364 INFO:tasks.rados.rados.0.smithi055.stdout:77: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2023-11-28T14:32:13.369 INFO:tasks.rados.rados.0.smithi055.stdout:77: writing smithi05583006-5 from 591613 to 1045775 tid 1 2023-11-28T14:32:13.375 INFO:tasks.rados.rados.0.smithi055.stdout:77: writing smithi05583006-5 from 1719614 to 2136967 tid 2 2023-11-28T14:32:13.381 INFO:tasks.rados.rados.0.smithi055.stdout:77: writing smithi05583006-5 from 2570780 to 2991613 tid 3 2023-11-28T14:32:13.381 INFO:tasks.rados.rados.0.smithi055.stdout:78: write oid 22 current snap is 1 2023-11-28T14:32:13.381 INFO:tasks.rados.rados.0.smithi055.stdout:78: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2023-11-28T14:32:13.388 INFO:tasks.rados.rados.0.smithi055.stdout:78: writing smithi05583006-22 from 608420 to 1137831 tid 1 2023-11-28T14:32:13.394 INFO:tasks.rados.rados.0.smithi055.stdout:78: writing smithi05583006-22 from 1778096 to 2339107 tid 2 2023-11-28T14:32:13.398 INFO:tasks.rados.rados.0.smithi055.stdout:78: writing smithi05583006-22 from 2881850 to 3008420 tid 3 2023-11-28T14:32:13.398 INFO:tasks.rados.rados.0.smithi055.stdout:79: write (excl) oid 34 current snap is 1 2023-11-28T14:32:13.398 INFO:tasks.rados.rados.0.smithi055.stdout:79: seq_num 61 ranges {625227=721013,1952931=704669} 2023-11-28T14:32:13.405 INFO:tasks.rados.rados.0.smithi055.stdout:79: writing smithi05583006-34 from 625227 to 1346240 tid 1 2023-11-28T14:32:13.415 INFO:tasks.rados.rados.0.smithi055.stdout:79: writing smithi05583006-34 from 1952931 to 2657600 tid 2 2023-11-28T14:32:13.415 INFO:tasks.rados.rados.0.smithi055.stdout:80: write (excl) oid 13 current snap is 1 2023-11-28T14:32:13.415 INFO:tasks.rados.rados.0.smithi055.stdout:80: seq_num 62 ranges {642034=796262,1895060=564680} 2023-11-28T14:32:13.423 INFO:tasks.rados.rados.0.smithi055.stdout:80: writing smithi05583006-13 from 642034 to 1438296 tid 1 2023-11-28T14:32:13.429 INFO:tasks.rados.rados.0.smithi055.stdout:80: writing smithi05583006-13 from 1895060 to 2459740 tid 2 2023-11-28T14:32:13.429 INFO:tasks.rados.rados.0.smithi055.stdout:81: delete oid 14 current snap is 1 2023-11-28T14:32:13.429 INFO:tasks.rados.rados.0.smithi055.stdout:68: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-28T14:32:13.520 INFO:tasks.rados.rados.0.smithi055.stdout:67: expect (ObjNum 25 snap 0 seq_num 25) 2023-11-28T14:32:13.622 INFO:tasks.rados.rados.0.smithi055.stdout:69: finishing write tid 1 to smithi05583006-47 2023-11-28T14:32:13.622 INFO:tasks.rados.rados.0.smithi055.stdout:67: done (15 left) 2023-11-28T14:32:13.623 INFO:tasks.rados.rados.0.smithi055.stdout:68: done (14 left) 2023-11-28T14:32:13.623 INFO:tasks.rados.rados.0.smithi055.stdout:81: done (13 left) 2023-11-28T14:32:13.623 INFO:tasks.rados.rados.0.smithi055.stdout:82: write oid 28 current snap is 1 2023-11-28T14:32:13.623 INFO:tasks.rados.rados.0.smithi055.stdout:82: seq_num 64 ranges {675648=546760,2012024=568349} 2023-11-28T14:32:13.629 INFO:tasks.rados.rados.0.smithi055.stdout:82: writing smithi05583006-28 from 675648 to 1222408 tid 1 2023-11-28T14:32:13.637 INFO:tasks.rados.rados.0.smithi055.stdout:82: writing smithi05583006-28 from 2012024 to 2580373 tid 2 2023-11-28T14:32:13.637 INFO:tasks.rados.rados.0.smithi055.stdout:83: write oid 4 current snap is 1 2023-11-28T14:32:13.637 INFO:tasks.rados.rados.0.smithi055.stdout:83: seq_num 65 ranges {692455=622009,2070506=712007} 2023-11-28T14:32:13.644 INFO:tasks.rados.rados.0.smithi055.stdout:83: writing smithi05583006-4 from 692455 to 1314464 tid 1 2023-11-28T14:32:13.653 INFO:tasks.rados.rados.0.smithi055.stdout:83: writing smithi05583006-4 from 2070506 to 2782513 tid 2 2023-11-28T14:32:13.654 INFO:tasks.rados.rados.0.smithi055.stdout:84: write oid 31 current snap is 1 2023-11-28T14:32:13.654 INFO:tasks.rados.rados.0.smithi055.stdout:84: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2023-11-28T14:32:13.661 INFO:tasks.rados.rados.0.smithi055.stdout:84: writing smithi05583006-31 from 709262 to 1406520 tid 1 2023-11-28T14:32:13.668 INFO:tasks.rados.rados.0.smithi055.stdout:84: writing smithi05583006-31 from 2012635 to 2584653 tid 2 2023-11-28T14:32:13.671 INFO:tasks.rados.rados.0.smithi055.stdout:84: writing smithi05583006-31 from 3046388 to 3109262 tid 3 2023-11-28T14:32:13.671 INFO:tasks.rados.rados.0.smithi055.stdout: waiting on 16 2023-11-28T14:32:13.671 INFO:tasks.rados.rados.0.smithi055.stdout:70: expect (ObjNum 39 snap 0 seq_num 39) 2023-11-28T14:32:13.832 INFO:tasks.rados.rados.0.smithi055.stdout:71: expect (ObjNum 35 snap 0 seq_num 35) 2023-11-28T14:32:13.887 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:32:13.953 INFO:tasks.rados.rados.0.smithi055.stdout:66: finishing write tid 1 to smithi05583006-33 2023-11-28T14:32:13.953 INFO:tasks.rados.rados.0.smithi055.stdout:75: expect (ObjNum 8 snap 0 seq_num 8) 2023-11-28T14:32:13.997 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:13.995+0000 7f79bcd00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:14.000 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:13.999+0000 7f79bcd00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:14.048 INFO:tasks.rados.rados.0.smithi055.stdout:66: finishing write tid 2 to smithi05583006-33 2023-11-28T14:32:14.048 INFO:tasks.rados.rados.0.smithi055.stdout:72: finishing write tid 1 to smithi05583006-17 2023-11-28T14:32:14.048 INFO:tasks.rados.rados.0.smithi055.stdout:76: expect (ObjNum 26 snap 0 seq_num 26) 2023-11-28T14:32:14.141 INFO:tasks.rados.rados.0.smithi055.stdout:74: finishing write tid 1 to smithi05583006-12 2023-11-28T14:32:14.141 INFO:tasks.rados.rados.0.smithi055.stdout:78: finishing write tid 1 to smithi05583006-22 2023-11-28T14:32:14.141 INFO:tasks.rados.rados.0.smithi055.stdout:69: finishing write tid 2 to smithi05583006-47 2023-11-28T14:32:14.142 INFO:tasks.rados.rados.0.smithi055.stdout:69: finishing write tid 3 to smithi05583006-47 2023-11-28T14:32:14.142 INFO:tasks.rados.rados.0.smithi055.stdout:72: finishing write tid 2 to smithi05583006-17 2023-11-28T14:32:14.142 INFO:tasks.rados.rados.0.smithi055.stdout:66: finishing write tid 3 to smithi05583006-33 2023-11-28T14:32:14.142 INFO:tasks.rados.rados.0.smithi055.stdout:74: finishing write tid 2 to smithi05583006-12 2023-11-28T14:32:14.142 INFO:tasks.rados.rados.0.smithi055.stdout:78: finishing write tid 2 to smithi05583006-22 2023-11-28T14:32:14.142 INFO:tasks.rados.rados.0.smithi055.stdout:73: finishing write tid 1 to smithi05583006-49 2023-11-28T14:32:14.142 INFO:tasks.rados.rados.0.smithi055.stdout:70: done (15 left) 2023-11-28T14:32:14.142 INFO:tasks.rados.rados.0.smithi055.stdout:71: done (14 left) 2023-11-28T14:32:14.142 INFO:tasks.rados.rados.0.smithi055.stdout:75: done (13 left) 2023-11-28T14:32:14.142 INFO:tasks.rados.rados.0.smithi055.stdout:76: done (12 left) 2023-11-28T14:32:14.142 INFO:tasks.rados.rados.0.smithi055.stdout:85: write oid 2 current snap is 1 2023-11-28T14:32:14.143 INFO:tasks.rados.rados.0.smithi055.stdout:85: seq_num 67 ranges {726069=772507,2071117=715676} 2023-11-28T14:32:14.149 INFO:tasks.rados.rados.0.smithi055.stdout:85: writing smithi05583006-2 from 726069 to 1498576 tid 1 2023-11-28T14:32:14.160 INFO:tasks.rados.rados.0.smithi055.stdout:85: writing smithi05583006-2 from 2071117 to 2786793 tid 2 2023-11-28T14:32:14.160 INFO:tasks.rados.rados.0.smithi055.stdout:86: delete oid 44 current snap is 1 2023-11-28T14:32:14.161 INFO:tasks.rados.rados.0.smithi055.stdout:72: finishing write tid 3 to smithi05583006-17 2023-11-28T14:32:14.161 INFO:tasks.rados.rados.0.smithi055.stdout:66: finishing write tid 4 to smithi05583006-33 2023-11-28T14:32:14.161 INFO:tasks.rados.rados.0.smithi055.stdout:69: finishing write tid 4 to smithi05583006-47 2023-11-28T14:32:14.161 INFO:tasks.rados.rados.0.smithi055.stdout:66: finishing write tid 5 to smithi05583006-33 2023-11-28T14:32:14.161 INFO:tasks.rados.rados.0.smithi055.stdout:66: oid 33 updating version 0 to 32 2023-11-28T14:32:14.161 INFO:tasks.rados.rados.0.smithi055.stdout:66: oid 33 version 32 is already newer than 30 2023-11-28T14:32:14.161 INFO:tasks.rados.rados.0.smithi055.stdout:66: oid 33 version 32 is already newer than 30 2023-11-28T14:32:14.161 INFO:tasks.rados.rados.0.smithi055.stdout:66: oid 33 version 32 is already newer than 31 2023-11-28T14:32:14.161 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 33 v 32 (ObjNum 54 snap 1 seq_num 54) dirty exists 2023-11-28T14:32:14.161 INFO:tasks.rados.rados.0.smithi055.stdout:66: left oid 33 (ObjNum 54 snap 1 seq_num 54) 2023-11-28T14:32:14.161 INFO:tasks.rados.rados.0.smithi055.stdout:74: finishing write tid 3 to smithi05583006-12 2023-11-28T14:32:14.161 INFO:tasks.rados.rados.0.smithi055.stdout:74: finishing write tid 4 to smithi05583006-12 2023-11-28T14:32:14.161 INFO:tasks.rados.rados.0.smithi055.stdout:74: oid 12 updating version 0 to 10 2023-11-28T14:32:14.162 INFO:tasks.rados.rados.0.smithi055.stdout:74: oid 12 version 10 is already newer than 9 2023-11-28T14:32:14.162 INFO:tasks.rados.rados.0.smithi055.stdout:74: oid 12 version 10 is already newer than 8 2023-11-28T14:32:14.162 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 12 v 10 (ObjNum 58 snap 1 seq_num 58) dirty exists 2023-11-28T14:32:14.162 INFO:tasks.rados.rados.0.smithi055.stdout:74: left oid 12 (ObjNum 58 snap 1 seq_num 58) 2023-11-28T14:32:14.162 INFO:tasks.rados.rados.0.smithi055.stdout:69: finishing write tid 5 to smithi05583006-47 2023-11-28T14:32:14.162 INFO:tasks.rados.rados.0.smithi055.stdout:69: oid 47 updating version 0 to 16 2023-11-28T14:32:14.162 INFO:tasks.rados.rados.0.smithi055.stdout:69: oid 47 version 16 is already newer than 16 2023-11-28T14:32:14.162 INFO:tasks.rados.rados.0.smithi055.stdout:69: oid 47 updating version 16 to 17 2023-11-28T14:32:14.162 INFO:tasks.rados.rados.0.smithi055.stdout:69: oid 47 version 17 is already newer than 14 2023-11-28T14:32:14.162 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 47 v 17 (ObjNum 55 snap 1 seq_num 55) dirty exists 2023-11-28T14:32:14.162 INFO:tasks.rados.rados.0.smithi055.stdout:69: left oid 47 (ObjNum 55 snap 1 seq_num 55) 2023-11-28T14:32:14.162 INFO:tasks.rados.rados.0.smithi055.stdout:73: finishing write tid 2 to smithi05583006-49 2023-11-28T14:32:14.163 INFO:tasks.rados.rados.0.smithi055.stdout:72: finishing write tid 4 to smithi05583006-17 2023-11-28T14:32:14.163 INFO:tasks.rados.rados.0.smithi055.stdout:72: finishing write tid 5 to smithi05583006-17 2023-11-28T14:32:14.163 INFO:tasks.rados.rados.0.smithi055.stdout:72: oid 17 updating version 0 to 5 2023-11-28T14:32:14.163 INFO:tasks.rados.rados.0.smithi055.stdout:72: oid 17 updating version 5 to 6 2023-11-28T14:32:14.163 INFO:tasks.rados.rados.0.smithi055.stdout:72: oid 17 updating version 6 to 8 2023-11-28T14:32:14.163 INFO:tasks.rados.rados.0.smithi055.stdout:72: oid 17 version 8 is already newer than 7 2023-11-28T14:32:14.163 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 17 v 8 (ObjNum 56 snap 1 seq_num 56) dirty exists 2023-11-28T14:32:14.164 INFO:tasks.rados.rados.0.smithi055.stdout:72: left oid 17 (ObjNum 56 snap 1 seq_num 56) 2023-11-28T14:32:14.164 INFO:tasks.rados.rados.0.smithi055.stdout:78: finishing write tid 3 to smithi05583006-22 2023-11-28T14:32:14.164 INFO:tasks.rados.rados.0.smithi055.stdout:78: finishing write tid 4 to smithi05583006-22 2023-11-28T14:32:14.164 INFO:tasks.rados.rados.0.smithi055.stdout:78: finishing write tid 5 to smithi05583006-22 2023-11-28T14:32:14.164 INFO:tasks.rados.rados.0.smithi055.stdout:78: oid 22 updating version 0 to 30 2023-11-28T14:32:14.164 INFO:tasks.rados.rados.0.smithi055.stdout:78: oid 22 version 30 is already newer than 27 2023-11-28T14:32:14.164 INFO:tasks.rados.rados.0.smithi055.stdout:78: oid 22 version 30 is already newer than 30 2023-11-28T14:32:14.164 INFO:tasks.rados.rados.0.smithi055.stdout:78: oid 22 version 30 is already newer than 28 2023-11-28T14:32:14.165 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 22 v 30 (ObjNum 60 snap 1 seq_num 60) dirty exists 2023-11-28T14:32:14.165 INFO:tasks.rados.rados.0.smithi055.stdout:78: left oid 22 (ObjNum 60 snap 1 seq_num 60) 2023-11-28T14:32:14.165 INFO:tasks.rados.rados.0.smithi055.stdout:77: finishing write tid 1 to smithi05583006-5 2023-11-28T14:32:14.165 INFO:tasks.rados.rados.0.smithi055.stdout:73: finishing write tid 3 to smithi05583006-49 2023-11-28T14:32:14.165 INFO:tasks.rados.rados.0.smithi055.stdout:77: finishing write tid 2 to smithi05583006-5 2023-11-28T14:32:14.165 INFO:tasks.rados.rados.0.smithi055.stdout:73: finishing write tid 4 to smithi05583006-49 2023-11-28T14:32:14.165 INFO:tasks.rados.rados.0.smithi055.stdout:73: finishing write tid 5 to smithi05583006-49 2023-11-28T14:32:14.166 INFO:tasks.rados.rados.0.smithi055.stdout:73: oid 49 updating version 0 to 13 2023-11-28T14:32:14.166 INFO:tasks.rados.rados.0.smithi055.stdout:73: oid 49 updating version 13 to 14 2023-11-28T14:32:14.166 INFO:tasks.rados.rados.0.smithi055.stdout:73: oid 49 version 14 is already newer than 11 2023-11-28T14:32:14.166 INFO:tasks.rados.rados.0.smithi055.stdout:73: oid 49 version 14 is already newer than 12 2023-11-28T14:32:14.166 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 49 v 14 (ObjNum 57 snap 1 seq_num 57) dirty exists 2023-11-28T14:32:14.166 INFO:tasks.rados.rados.0.smithi055.stdout:73: left oid 49 (ObjNum 57 snap 1 seq_num 57) 2023-11-28T14:32:14.166 INFO:tasks.rados.rados.0.smithi055.stdout:79: finishing write tid 1 to smithi05583006-34 2023-11-28T14:32:14.167 INFO:tasks.rados.rados.0.smithi055.stdout:77: finishing write tid 3 to smithi05583006-5 2023-11-28T14:32:14.167 INFO:tasks.rados.rados.0.smithi055.stdout:79: finishing write tid 2 to smithi05583006-34 2023-11-28T14:32:14.167 INFO:tasks.rados.rados.0.smithi055.stdout:79: finishing write tid 3 to smithi05583006-34 2023-11-28T14:32:14.167 INFO:tasks.rados.rados.0.smithi055.stdout:79: finishing write tid 4 to smithi05583006-34 2023-11-28T14:32:14.167 INFO:tasks.rados.rados.0.smithi055.stdout:79: oid 34 updating version 0 to 16 2023-11-28T14:32:14.167 INFO:tasks.rados.rados.0.smithi055.stdout:79: oid 34 updating version 16 to 18 2023-11-28T14:32:14.168 INFO:tasks.rados.rados.0.smithi055.stdout:79: oid 34 version 18 is already newer than 18 2023-11-28T14:32:14.168 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 34 v 18 (ObjNum 61 snap 1 seq_num 61) dirty exists 2023-11-28T14:32:14.168 INFO:tasks.rados.rados.0.smithi055.stdout:79: left oid 34 (ObjNum 61 snap 1 seq_num 61) 2023-11-28T14:32:14.168 INFO:tasks.rados.rados.0.smithi055.stdout:84: finishing write tid 1 to smithi05583006-31 2023-11-28T14:32:14.168 INFO:tasks.rados.rados.0.smithi055.stdout:82: finishing write tid 1 to smithi05583006-28 2023-11-28T14:32:14.168 INFO:tasks.rados.rados.0.smithi055.stdout:77: finishing write tid 4 to smithi05583006-5 2023-11-28T14:32:14.168 INFO:tasks.rados.rados.0.smithi055.stdout:77: finishing write tid 5 to smithi05583006-5 2023-11-28T14:32:14.169 INFO:tasks.rados.rados.0.smithi055.stdout:77: oid 5 updating version 0 to 37 2023-11-28T14:32:14.169 INFO:tasks.rados.rados.0.smithi055.stdout:77: oid 5 version 37 is already newer than 35 2023-11-28T14:32:14.169 INFO:tasks.rados.rados.0.smithi055.stdout:77: oid 5 version 37 is already newer than 36 2023-11-28T14:32:14.169 INFO:tasks.rados.rados.0.smithi055.stdout:77: oid 5 version 37 is already newer than 34 2023-11-28T14:32:14.169 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 5 v 37 (ObjNum 59 snap 1 seq_num 59) dirty exists 2023-11-28T14:32:14.169 INFO:tasks.rados.rados.0.smithi055.stdout:77: left oid 5 (ObjNum 59 snap 1 seq_num 59) 2023-11-28T14:32:14.169 INFO:tasks.rados.rados.0.smithi055.stdout:84: finishing write tid 2 to smithi05583006-31 2023-11-28T14:32:14.169 INFO:tasks.rados.rados.0.smithi055.stdout:82: finishing write tid 2 to smithi05583006-28 2023-11-28T14:32:14.170 INFO:tasks.rados.rados.0.smithi055.stdout:82: finishing write tid 3 to smithi05583006-28 2023-11-28T14:32:14.170 INFO:tasks.rados.rados.0.smithi055.stdout:82: finishing write tid 4 to smithi05583006-28 2023-11-28T14:32:14.170 INFO:tasks.rados.rados.0.smithi055.stdout:82: oid 28 updating version 0 to 13 2023-11-28T14:32:14.170 INFO:tasks.rados.rados.0.smithi055.stdout:82: oid 28 version 13 is already newer than 11 2023-11-28T14:32:14.170 INFO:tasks.rados.rados.0.smithi055.stdout:82: oid 28 version 13 is already newer than 13 2023-11-28T14:32:14.170 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 28 v 13 (ObjNum 64 snap 1 seq_num 64) dirty exists 2023-11-28T14:32:14.170 INFO:tasks.rados.rados.0.smithi055.stdout:82: left oid 28 (ObjNum 64 snap 1 seq_num 64) 2023-11-28T14:32:14.170 INFO:tasks.rados.rados.0.smithi055.stdout:83: finishing write tid 1 to smithi05583006-4 2023-11-28T14:32:14.170 INFO:tasks.rados.rados.0.smithi055.stdout:80: finishing write tid 1 to smithi05583006-13 2023-11-28T14:32:14.171 INFO:tasks.rados.rados.0.smithi055.stdout:84: finishing write tid 3 to smithi05583006-31 2023-11-28T14:32:14.171 INFO:tasks.rados.rados.0.smithi055.stdout:83: finishing write tid 2 to smithi05583006-4 2023-11-28T14:32:14.171 INFO:tasks.rados.rados.0.smithi055.stdout:80: finishing write tid 2 to smithi05583006-13 2023-11-28T14:32:14.171 INFO:tasks.rados.rados.0.smithi055.stdout:84: finishing write tid 4 to smithi05583006-31 2023-11-28T14:32:14.171 INFO:tasks.rados.rados.0.smithi055.stdout:84: finishing write tid 5 to smithi05583006-31 2023-11-28T14:32:14.171 INFO:tasks.rados.rados.0.smithi055.stdout:84: oid 31 updating version 0 to 11 2023-11-28T14:32:14.171 INFO:tasks.rados.rados.0.smithi055.stdout:84: oid 31 version 11 is already newer than 10 2023-11-28T14:32:14.171 INFO:tasks.rados.rados.0.smithi055.stdout:84: oid 31 version 11 is already newer than 8 2023-11-28T14:32:14.171 INFO:tasks.rados.rados.0.smithi055.stdout:84: oid 31 version 11 is already newer than 9 2023-11-28T14:32:14.171 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 31 v 11 (ObjNum 66 snap 1 seq_num 66) dirty exists 2023-11-28T14:32:14.171 INFO:tasks.rados.rados.0.smithi055.stdout:84: left oid 31 (ObjNum 66 snap 1 seq_num 66) 2023-11-28T14:32:14.171 INFO:tasks.rados.rados.0.smithi055.stdout:83: finishing write tid 3 to smithi05583006-4 2023-11-28T14:32:14.171 INFO:tasks.rados.rados.0.smithi055.stdout:83: finishing write tid 4 to smithi05583006-4 2023-11-28T14:32:14.172 INFO:tasks.rados.rados.0.smithi055.stdout:83: oid 4 updating version 0 to 8 2023-11-28T14:32:14.172 INFO:tasks.rados.rados.0.smithi055.stdout:83: oid 4 updating version 8 to 9 2023-11-28T14:32:14.172 INFO:tasks.rados.rados.0.smithi055.stdout:83: oid 4 updating version 9 to 10 2023-11-28T14:32:14.172 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 4 v 10 (ObjNum 65 snap 1 seq_num 65) dirty exists 2023-11-28T14:32:14.172 INFO:tasks.rados.rados.0.smithi055.stdout:83: left oid 4 (ObjNum 65 snap 1 seq_num 65) 2023-11-28T14:32:14.172 INFO:tasks.rados.rados.0.smithi055.stdout:80: finishing write tid 3 to smithi05583006-13 2023-11-28T14:32:14.172 INFO:tasks.rados.rados.0.smithi055.stdout:80: finishing write tid 4 to smithi05583006-13 2023-11-28T14:32:14.172 INFO:tasks.rados.rados.0.smithi055.stdout:80: oid 13 updating version 0 to 40 2023-11-28T14:32:14.172 INFO:tasks.rados.rados.0.smithi055.stdout:80: oid 13 version 40 is already newer than 39 2023-11-28T14:32:14.172 INFO:tasks.rados.rados.0.smithi055.stdout:80: oid 13 updating version 40 to 41 2023-11-28T14:32:14.172 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 13 v 41 (ObjNum 62 snap 1 seq_num 62) dirty exists 2023-11-28T14:32:14.172 INFO:tasks.rados.rados.0.smithi055.stdout:80: left oid 13 (ObjNum 62 snap 1 seq_num 62) 2023-11-28T14:32:14.178 INFO:tasks.rados.rados.0.smithi055.stdout:85: finishing write tid 1 to smithi05583006-2 2023-11-28T14:32:14.192 INFO:tasks.rados.rados.0.smithi055.stdout:85: finishing write tid 2 to smithi05583006-2 2023-11-28T14:32:14.199 INFO:tasks.rados.rados.0.smithi055.stdout:85: finishing write tid 3 to smithi05583006-2 2023-11-28T14:32:14.202 INFO:tasks.rados.rados.0.smithi055.stdout:85: finishing write tid 4 to smithi05583006-2 2023-11-28T14:32:14.202 INFO:tasks.rados.rados.0.smithi055.stdout:85: oid 2 updating version 0 to 34 2023-11-28T14:32:14.202 INFO:tasks.rados.rados.0.smithi055.stdout:85: oid 2 updating version 34 to 35 2023-11-28T14:32:14.202 INFO:tasks.rados.rados.0.smithi055.stdout:85: oid 2 updating version 35 to 36 2023-11-28T14:32:14.202 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 2 v 36 (ObjNum 67 snap 1 seq_num 67) dirty exists 2023-11-28T14:32:14.203 INFO:tasks.rados.rados.0.smithi055.stdout:85: left oid 2 (ObjNum 67 snap 1 seq_num 67) 2023-11-28T14:32:14.215 INFO:tasks.rados.rados.0.smithi055.stdout:66: done (13 left) 2023-11-28T14:32:14.215 INFO:tasks.rados.rados.0.smithi055.stdout:69: done (12 left) 2023-11-28T14:32:14.215 INFO:tasks.rados.rados.0.smithi055.stdout:72: done (11 left) 2023-11-28T14:32:14.215 INFO:tasks.rados.rados.0.smithi055.stdout:73: done (10 left) 2023-11-28T14:32:14.215 INFO:tasks.rados.rados.0.smithi055.stdout:74: done (9 left) 2023-11-28T14:32:14.215 INFO:tasks.rados.rados.0.smithi055.stdout:77: done (8 left) 2023-11-28T14:32:14.216 INFO:tasks.rados.rados.0.smithi055.stdout:78: done (7 left) 2023-11-28T14:32:14.216 INFO:tasks.rados.rados.0.smithi055.stdout:79: done (6 left) 2023-11-28T14:32:14.216 INFO:tasks.rados.rados.0.smithi055.stdout:80: done (5 left) 2023-11-28T14:32:14.216 INFO:tasks.rados.rados.0.smithi055.stdout:82: done (4 left) 2023-11-28T14:32:14.216 INFO:tasks.rados.rados.0.smithi055.stdout:83: done (3 left) 2023-11-28T14:32:14.216 INFO:tasks.rados.rados.0.smithi055.stdout:84: done (2 left) 2023-11-28T14:32:14.216 INFO:tasks.rados.rados.0.smithi055.stdout:85: done (1 left) 2023-11-28T14:32:14.216 INFO:tasks.rados.rados.0.smithi055.stdout:86: done (0 left) 2023-11-28T14:32:14.217 INFO:tasks.rados.rados.0.smithi055.stdout:87: write (excl) oid 44 current snap is 1 2023-11-28T14:32:14.217 INFO:tasks.rados.rados.0.smithi055.stdout:87: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2023-11-28T14:32:14.222 INFO:tasks.rados.rados.0.smithi055.stdout:87: writing smithi05583006-44 from 759683 to 1399041 tid 1 2023-11-28T14:32:14.230 INFO:tasks.rados.rados.0.smithi055.stdout:87: writing smithi05583006-44 from 1904434 to 2623779 tid 2 2023-11-28T14:32:14.234 INFO:tasks.rados.rados.0.smithi055.stdout:87: writing smithi05583006-44 from 3128657 to 3159683 tid 3 2023-11-28T14:32:14.234 INFO:tasks.rados.rados.0.smithi055.stdout:88: write oid 22 current snap is 1 2023-11-28T14:32:14.234 INFO:tasks.rados.rados.0.smithi055.stdout:88: seq_num 70 ranges {776490=714607,2246563=579356} 2023-11-28T14:32:14.244 INFO:tasks.rados.rados.0.smithi055.stdout:88: writing smithi05583006-22 from 776490 to 1491097 tid 1 2023-11-28T14:32:14.252 INFO:tasks.rados.rados.0.smithi055.stdout:88: writing smithi05583006-22 from 2246563 to 2825919 tid 2 2023-11-28T14:32:14.252 INFO:tasks.rados.rados.0.smithi055.stdout:89: read oid 23 snap -1 2023-11-28T14:32:14.252 INFO:tasks.rados.rados.0.smithi055.stdout:89: expect (ObjNum 22 snap 0 seq_num 22) 2023-11-28T14:32:14.252 INFO:tasks.rados.rados.0.smithi055.stdout:90: write (excl) oid 4 current snap is 1 2023-11-28T14:32:14.252 INFO:tasks.rados.rados.0.smithi055.stdout:87: finishing write tid 1 to smithi05583006-44 2023-11-28T14:32:14.252 INFO:tasks.rados.rados.0.smithi055.stdout:90: seq_num 71 ranges {793297=789856,2305045=723014} 2023-11-28T14:32:14.262 INFO:tasks.rados.rados.0.smithi055.stdout:90: writing smithi05583006-4 from 793297 to 1583153 tid 1 2023-11-28T14:32:14.272 INFO:tasks.rados.rados.0.smithi055.stdout:90: writing smithi05583006-4 from 2305045 to 3028059 tid 2 2023-11-28T14:32:14.272 INFO:tasks.rados.rados.0.smithi055.stdout:91: write oid 16 current snap is 1 2023-11-28T14:32:14.272 INFO:tasks.rados.rados.0.smithi055.stdout:91: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2023-11-28T14:32:14.277 INFO:tasks.rados.rados.0.smithi055.stdout:91: writing smithi05583006-16 from 410104 to 875209 tid 1 2023-11-28T14:32:14.285 INFO:tasks.rados.rados.0.smithi055.stdout:91: writing smithi05583006-16 from 1563527 to 2146552 tid 2 2023-11-28T14:32:14.291 INFO:tasks.rados.rados.0.smithi055.stdout:91: writing smithi05583006-16 from 2810926 to 3210104 tid 3 2023-11-28T14:32:14.291 INFO:tasks.rados.rados.0.smithi055.stdout:92: read oid 28 snap -1 2023-11-28T14:32:14.291 INFO:tasks.rados.rados.0.smithi055.stdout:92: expect (ObjNum 64 snap 1 seq_num 64) 2023-11-28T14:32:14.292 INFO:tasks.rados.rados.0.smithi055.stdout:93: snap_create 2023-11-28T14:32:14.292 INFO:tasks.rados.rados.0.smithi055.stdout:87: finishing write tid 2 to smithi05583006-44 2023-11-28T14:32:14.292 INFO:tasks.rados.rados.0.smithi055.stdout:89: expect (ObjNum 22 snap 0 seq_num 22) 2023-11-28T14:32:14.390 INFO:tasks.rados.rados.0.smithi055.stdout:87: finishing write tid 3 to smithi05583006-44 2023-11-28T14:32:14.391 INFO:tasks.rados.rados.0.smithi055.stdout:90: finishing write tid 1 to smithi05583006-4 2023-11-28T14:32:14.391 INFO:tasks.rados.rados.0.smithi055.stdout:91: finishing write tid 1 to smithi05583006-16 2023-11-28T14:32:14.391 INFO:tasks.rados.rados.0.smithi055.stdout:87: finishing write tid 4 to smithi05583006-44 2023-11-28T14:32:14.391 INFO:tasks.rados.rados.0.smithi055.stdout:87: finishing write tid 5 to smithi05583006-44 2023-11-28T14:32:14.391 INFO:tasks.rados.rados.0.smithi055.stdout:87: oid 44 updating version 0 to 41 2023-11-28T14:32:14.391 INFO:tasks.rados.rados.0.smithi055.stdout:87: oid 44 updating version 41 to 42 2023-11-28T14:32:14.391 INFO:tasks.rados.rados.0.smithi055.stdout:87: oid 44 version 42 is already newer than 39 2023-11-28T14:32:14.391 INFO:tasks.rados.rados.0.smithi055.stdout:87: oid 44 version 42 is already newer than 40 2023-11-28T14:32:14.391 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 44 v 42 (ObjNum 69 snap 1 seq_num 69) dirty exists 2023-11-28T14:32:14.391 INFO:tasks.rados.rados.0.smithi055.stdout:87: left oid 44 (ObjNum 69 snap 1 seq_num 69) 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:90: finishing write tid 2 to smithi05583006-4 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:88: finishing write tid 1 to smithi05583006-22 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:90: finishing write tid 3 to smithi05583006-4 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:88: finishing write tid 2 to smithi05583006-22 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:88: finishing write tid 3 to smithi05583006-22 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:88: finishing write tid 4 to smithi05583006-22 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:88: oid 22 updating version 0 to 45 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:88: oid 22 version 45 is already newer than 45 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:88: oid 22 version 45 is already newer than 45 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 22 v 45 (ObjNum 70 snap 1 seq_num 70) dirty exists 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:88: left oid 22 (ObjNum 70 snap 1 seq_num 70) 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:90: finishing write tid 4 to smithi05583006-4 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:90: oid 4 updating version 0 to 11 2023-11-28T14:32:14.392 INFO:tasks.rados.rados.0.smithi055.stdout:90: oid 4 updating version 11 to 12 2023-11-28T14:32:14.393 INFO:tasks.rados.rados.0.smithi055.stdout:90: oid 4 updating version 12 to 13 2023-11-28T14:32:14.393 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 4 v 13 (ObjNum 71 snap 1 seq_num 71) dirty exists 2023-11-28T14:32:14.393 INFO:tasks.rados.rados.0.smithi055.stdout:90: left oid 4 (ObjNum 71 snap 1 seq_num 71) 2023-11-28T14:32:14.393 INFO:tasks.rados.rados.0.smithi055.stdout:92: expect (ObjNum 64 snap 1 seq_num 64) 2023-11-28T14:32:14.550 INFO:tasks.rados.rados.0.smithi055.stdout:91: finishing write tid 2 to smithi05583006-16 2023-11-28T14:32:14.550 INFO:tasks.rados.rados.0.smithi055.stdout:91: finishing write tid 3 to smithi05583006-16 2023-11-28T14:32:14.550 INFO:tasks.rados.rados.0.smithi055.stdout:91: finishing write tid 4 to smithi05583006-16 2023-11-28T14:32:14.550 INFO:tasks.rados.rados.0.smithi055.stdout:91: finishing write tid 5 to smithi05583006-16 2023-11-28T14:32:14.550 INFO:tasks.rados.rados.0.smithi055.stdout:91: oid 16 updating version 0 to 43 2023-11-28T14:32:14.550 INFO:tasks.rados.rados.0.smithi055.stdout:91: oid 16 updating version 43 to 44 2023-11-28T14:32:14.550 INFO:tasks.rados.rados.0.smithi055.stdout:91: oid 16 updating version 44 to 46 2023-11-28T14:32:14.550 INFO:tasks.rados.rados.0.smithi055.stdout:91: oid 16 version 46 is already newer than 45 2023-11-28T14:32:14.551 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 16 v 46 (ObjNum 72 snap 1 seq_num 72) dirty exists 2023-11-28T14:32:14.551 INFO:tasks.rados.rados.0.smithi055.stdout:91: left oid 16 (ObjNum 72 snap 1 seq_num 72) 2023-11-28T14:32:15.171 INFO:tasks.rados.rados.0.smithi055.stdout:87: done (6 left) 2023-11-28T14:32:15.171 INFO:tasks.rados.rados.0.smithi055.stdout:88: done (5 left) 2023-11-28T14:32:15.171 INFO:tasks.rados.rados.0.smithi055.stdout:89: done (4 left) 2023-11-28T14:32:15.171 INFO:tasks.rados.rados.0.smithi055.stdout:90: done (3 left) 2023-11-28T14:32:15.172 INFO:tasks.rados.rados.0.smithi055.stdout:91: done (2 left) 2023-11-28T14:32:15.172 INFO:tasks.rados.rados.0.smithi055.stdout:92: done (1 left) 2023-11-28T14:32:15.172 INFO:tasks.rados.rados.0.smithi055.stdout:93: done (0 left) 2023-11-28T14:32:15.172 INFO:tasks.rados.rados.0.smithi055.stdout:94: snap_remove snap 1 2023-11-28T14:32:16.080 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:32:16.173 INFO:tasks.rados.rados.0.smithi055.stdout:94: done (0 left) 2023-11-28T14:32:16.174 INFO:tasks.rados.rados.0.smithi055.stdout:95: write oid 31 current snap is 2 2023-11-28T14:32:16.174 INFO:tasks.rados.rados.0.smithi055.stdout:95: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2023-11-28T14:32:16.180 INFO:tasks.rados.rados.0.smithi055.stdout:95: writing smithi05583006-31 from 426911 to 967265 tid 1 2023-11-28T14:32:16.190 INFO:tasks.rados.rados.0.smithi055.stdout:95: writing smithi05583006-31 from 1622009 to 2348692 tid 2 2023-11-28T14:32:16.196 INFO:tasks.rados.rados.0.smithi055.stdout:95: writing smithi05583006-31 from 3121996 to 3226911 tid 3 2023-11-28T14:32:16.196 INFO:tasks.rados.rados.0.smithi055.stdout:96: read oid 16 snap -1 2023-11-28T14:32:16.196 INFO:tasks.rados.rados.0.smithi055.stdout:96: expect (ObjNum 72 snap 1 seq_num 72) 2023-11-28T14:32:16.196 INFO:tasks.rados.rados.0.smithi055.stdout:97: snap_remove snap 0 2023-11-28T14:32:16.221 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:16.219+0000 7f682091e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:16.225 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:16.224+0000 7f682091e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:17.189 INFO:tasks.rados.rados.0.smithi055.stdout:97: done (2 left) 2023-11-28T14:32:17.189 INFO:tasks.rados.rados.0.smithi055.stdout:98: delete oid 12 current snap is 2 2023-11-28T14:32:17.189 INFO:tasks.rados.rados.0.smithi055.stdout:95: finishing write tid 1 to smithi05583006-31 2023-11-28T14:32:17.189 INFO:tasks.rados.rados.0.smithi055.stdout:96: expect (ObjNum 72 snap 1 seq_num 72) 2023-11-28T14:32:17.364 INFO:tasks.rados.rados.0.smithi055.stdout:95: finishing write tid 2 to smithi05583006-31 2023-11-28T14:32:17.365 INFO:tasks.rados.rados.0.smithi055.stdout:95: finishing write tid 3 to smithi05583006-31 2023-11-28T14:32:17.365 INFO:tasks.rados.rados.0.smithi055.stdout:96: done (2 left) 2023-11-28T14:32:17.365 INFO:tasks.rados.rados.0.smithi055.stdout:98: done (1 left) 2023-11-28T14:32:17.365 INFO:tasks.rados.rados.0.smithi055.stdout:99: read oid 9 snap -1 2023-11-28T14:32:17.365 INFO:tasks.rados.rados.0.smithi055.stdout:99: expect (ObjNum 8 snap 0 seq_num 8) 2023-11-28T14:32:17.365 INFO:tasks.rados.rados.0.smithi055.stdout:100: read oid 47 snap -1 2023-11-28T14:32:17.365 INFO:tasks.rados.rados.0.smithi055.stdout:100: expect (ObjNum 55 snap 1 seq_num 55) 2023-11-28T14:32:17.365 INFO:tasks.rados.rados.0.smithi055.stdout:101: write oid 5 current snap is 2 2023-11-28T14:32:17.365 INFO:tasks.rados.rados.0.smithi055.stdout:95: finishing write tid 4 to smithi05583006-31 2023-11-28T14:32:17.365 INFO:tasks.rados.rados.0.smithi055.stdout:95: finishing write tid 5 to smithi05583006-31 2023-11-28T14:32:17.366 INFO:tasks.rados.rados.0.smithi055.stdout:95: oid 31 updating version 0 to 15 2023-11-28T14:32:17.366 INFO:tasks.rados.rados.0.smithi055.stdout:95: oid 31 version 15 is already newer than 14 2023-11-28T14:32:17.366 INFO:tasks.rados.rados.0.smithi055.stdout:95: oid 31 version 15 is already newer than 12 2023-11-28T14:32:17.366 INFO:tasks.rados.rados.0.smithi055.stdout:95: oid 31 version 15 is already newer than 13 2023-11-28T14:32:17.366 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 31 v 15 (ObjNum 73 snap 2 seq_num 73) dirty exists 2023-11-28T14:32:17.366 INFO:tasks.rados.rados.0.smithi055.stdout:95: left oid 31 (ObjNum 73 snap 2 seq_num 73) 2023-11-28T14:32:17.366 INFO:tasks.rados.rados.0.smithi055.stdout:101: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2023-11-28T14:32:17.371 INFO:tasks.rados.rados.0.smithi055.stdout:101: writing smithi05583006-5 from 460525 to 1151377 tid 1 2023-11-28T14:32:17.379 INFO:tasks.rados.rados.0.smithi055.stdout:101: writing smithi05583006-5 from 1622620 to 2352972 tid 2 2023-11-28T14:32:17.385 INFO:tasks.rados.rados.0.smithi055.stdout:101: writing smithi05583006-5 from 3060489 to 3260525 tid 3 2023-11-28T14:32:17.385 INFO:tasks.rados.rados.0.smithi055.stdout:95: done (3 left) 2023-11-28T14:32:17.385 INFO:tasks.rados.rados.0.smithi055.stdout:102: write oid 26 current snap is 2 2023-11-28T14:32:17.385 INFO:tasks.rados.rados.0.smithi055.stdout:102: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2023-11-28T14:32:17.392 INFO:tasks.rados.rados.0.smithi055.stdout:102: writing smithi05583006-26 from 477332 to 1243433 tid 1 2023-11-28T14:32:17.398 INFO:tasks.rados.rados.0.smithi055.stdout:102: writing smithi05583006-26 from 1681102 to 2155112 tid 2 2023-11-28T14:32:17.405 INFO:tasks.rados.rados.0.smithi055.stdout:102: writing smithi05583006-26 from 2687912 to 3277332 tid 3 2023-11-28T14:32:17.405 INFO:tasks.rados.rados.0.smithi055.stdout:103: read oid 39 snap -1 2023-11-28T14:32:17.405 INFO:tasks.rados.rados.0.smithi055.stdout:103: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-28T14:32:17.405 INFO:tasks.rados.rados.0.smithi055.stdout:104: read oid 29 snap -1 2023-11-28T14:32:17.405 INFO:tasks.rados.rados.0.smithi055.stdout:104: expect (ObjNum 28 snap 0 seq_num 28) 2023-11-28T14:32:17.405 INFO:tasks.rados.rados.0.smithi055.stdout:105: delete oid 13 current snap is 2 2023-11-28T14:32:17.412 INFO:tasks.rados.rados.0.smithi055.stdout:99: expect (ObjNum 8 snap 0 seq_num 8) 2023-11-28T14:32:17.539 INFO:tasks.rados.rados.0.smithi055.stdout:100: expect (ObjNum 55 snap 1 seq_num 55) 2023-11-28T14:32:17.730 INFO:tasks.rados.rados.0.smithi055.stdout:103: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-28T14:32:17.831 INFO:tasks.rados.rados.0.smithi055.stdout:104: expect (ObjNum 28 snap 0 seq_num 28) 2023-11-28T14:32:17.927 INFO:tasks.rados.rados.0.smithi055.stdout:102: finishing write tid 1 to smithi05583006-26 2023-11-28T14:32:17.928 INFO:tasks.rados.rados.0.smithi055.stdout:101: finishing write tid 1 to smithi05583006-5 2023-11-28T14:32:17.928 INFO:tasks.rados.rados.0.smithi055.stdout:101: finishing write tid 2 to smithi05583006-5 2023-11-28T14:32:17.928 INFO:tasks.rados.rados.0.smithi055.stdout:101: finishing write tid 3 to smithi05583006-5 2023-11-28T14:32:17.928 INFO:tasks.rados.rados.0.smithi055.stdout:102: finishing write tid 2 to smithi05583006-26 2023-11-28T14:32:17.928 INFO:tasks.rados.rados.0.smithi055.stdout:101: finishing write tid 4 to smithi05583006-5 2023-11-28T14:32:17.928 INFO:tasks.rados.rados.0.smithi055.stdout:101: finishing write tid 5 to smithi05583006-5 2023-11-28T14:32:17.928 INFO:tasks.rados.rados.0.smithi055.stdout:101: oid 5 updating version 0 to 53 2023-11-28T14:32:17.928 INFO:tasks.rados.rados.0.smithi055.stdout:101: oid 5 updating version 53 to 54 2023-11-28T14:32:17.928 INFO:tasks.rados.rados.0.smithi055.stdout:101: oid 5 version 54 is already newer than 52 2023-11-28T14:32:17.928 INFO:tasks.rados.rados.0.smithi055.stdout:101: oid 5 version 54 is already newer than 51 2023-11-28T14:32:17.928 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 5 v 54 (ObjNum 75 snap 2 seq_num 75) dirty exists 2023-11-28T14:32:17.928 INFO:tasks.rados.rados.0.smithi055.stdout:101: left oid 5 (ObjNum 75 snap 2 seq_num 75) 2023-11-28T14:32:17.929 INFO:tasks.rados.rados.0.smithi055.stdout:102: finishing write tid 3 to smithi05583006-26 2023-11-28T14:32:17.929 INFO:tasks.rados.rados.0.smithi055.stdout:102: finishing write tid 4 to smithi05583006-26 2023-11-28T14:32:17.929 INFO:tasks.rados.rados.0.smithi055.stdout:102: finishing write tid 5 to smithi05583006-26 2023-11-28T14:32:17.929 INFO:tasks.rados.rados.0.smithi055.stdout:102: oid 26 updating version 0 to 20 2023-11-28T14:32:17.929 INFO:tasks.rados.rados.0.smithi055.stdout:102: oid 26 updating version 20 to 21 2023-11-28T14:32:17.929 INFO:tasks.rados.rados.0.smithi055.stdout:102: oid 26 version 21 is already newer than 18 2023-11-28T14:32:17.929 INFO:tasks.rados.rados.0.smithi055.stdout:102: oid 26 version 21 is already newer than 19 2023-11-28T14:32:17.929 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 26 v 21 (ObjNum 76 snap 2 seq_num 76) dirty exists 2023-11-28T14:32:17.929 INFO:tasks.rados.rados.0.smithi055.stdout:102: left oid 26 (ObjNum 76 snap 2 seq_num 76) 2023-11-28T14:32:17.929 INFO:tasks.rados.rados.0.smithi055.stdout:99: done (6 left) 2023-11-28T14:32:17.929 INFO:tasks.rados.rados.0.smithi055.stdout:100: done (5 left) 2023-11-28T14:32:17.929 INFO:tasks.rados.rados.0.smithi055.stdout:101: done (4 left) 2023-11-28T14:32:17.930 INFO:tasks.rados.rados.0.smithi055.stdout:102: done (3 left) 2023-11-28T14:32:17.930 INFO:tasks.rados.rados.0.smithi055.stdout:103: done (2 left) 2023-11-28T14:32:17.930 INFO:tasks.rados.rados.0.smithi055.stdout:104: done (1 left) 2023-11-28T14:32:17.930 INFO:tasks.rados.rados.0.smithi055.stdout:105: done (0 left) 2023-11-28T14:32:17.930 INFO:tasks.rados.rados.0.smithi055.stdout:106: read oid 14 snap -1 2023-11-28T14:32:17.930 INFO:tasks.rados.rados.0.smithi055.stdout:106: expect deleted 2023-11-28T14:32:17.930 INFO:tasks.rados.rados.0.smithi055.stdout:107: delete oid 30 current snap is 2 2023-11-28T14:32:17.942 INFO:tasks.rados.rados.0.smithi055.stdout:106: done (1 left) 2023-11-28T14:32:17.943 INFO:tasks.rados.rados.0.smithi055.stdout:107: done (0 left) 2023-11-28T14:32:17.943 INFO:tasks.rados.rados.0.smithi055.stdout:108: read oid 3 snap -1 2023-11-28T14:32:17.943 INFO:tasks.rados.rados.0.smithi055.stdout:108: expect (ObjNum 2 snap 0 seq_num 2) 2023-11-28T14:32:17.943 INFO:tasks.rados.rados.0.smithi055.stdout:109: read oid 29 snap -1 2023-11-28T14:32:17.943 INFO:tasks.rados.rados.0.smithi055.stdout:109: expect (ObjNum 28 snap 0 seq_num 28) 2023-11-28T14:32:17.943 INFO:tasks.rados.rados.0.smithi055.stdout:110: write oid 49 current snap is 2 2023-11-28T14:32:17.944 INFO:tasks.rados.rados.0.smithi055.stdout:110: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2023-11-28T14:32:17.950 INFO:tasks.rados.rados.0.smithi055.stdout:110: writing smithi05583006-49 from 527753 to 1235954 tid 1 2023-11-28T14:32:17.959 INFO:tasks.rados.rados.0.smithi055.stdout:110: writing smithi05583006-49 from 1856548 to 2594238 tid 2 2023-11-28T14:32:17.963 INFO:tasks.rados.rados.0.smithi055.stdout:110: writing smithi05583006-49 from 3286534 to 3327753 tid 3 2023-11-28T14:32:17.964 INFO:tasks.rados.rados.0.smithi055.stdout:111: delete oid 34 current snap is 2 2023-11-28T14:32:17.964 INFO:tasks.rados.rados.0.smithi055.stdout:109: expect (ObjNum 28 snap 0 seq_num 28) 2023-11-28T14:32:18.017 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:32:18.017 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:32:18.017 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-11-28T14:32:18.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-11-28T14:32:18.060 INFO:tasks.rados.rados.0.smithi055.stdout:108: expect (ObjNum 2 snap 0 seq_num 2) 2023-11-28T14:32:18.131 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:18.126+0000 7f819726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:18.134 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:18.133+0000 7f819726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:18.139 INFO:tasks.rados.rados.0.smithi055.stdout:110: finishing write tid 1 to smithi05583006-49 2023-11-28T14:32:18.139 INFO:tasks.rados.rados.0.smithi055.stdout:110: finishing write tid 2 to smithi05583006-49 2023-11-28T14:32:18.139 INFO:tasks.rados.rados.0.smithi055.stdout:110: finishing write tid 3 to smithi05583006-49 2023-11-28T14:32:18.139 INFO:tasks.rados.rados.0.smithi055.stdout:108: done (3 left) 2023-11-28T14:32:18.139 INFO:tasks.rados.rados.0.smithi055.stdout:109: done (2 left) 2023-11-28T14:32:18.139 INFO:tasks.rados.rados.0.smithi055.stdout:111: done (1 left) 2023-11-28T14:32:18.139 INFO:tasks.rados.rados.0.smithi055.stdout:112: read oid 22 snap -1 2023-11-28T14:32:18.139 INFO:tasks.rados.rados.0.smithi055.stdout:112: expect (ObjNum 70 snap 1 seq_num 70) 2023-11-28T14:32:18.139 INFO:tasks.rados.rados.0.smithi055.stdout:113: write oid 2 current snap is 2 2023-11-28T14:32:18.140 INFO:tasks.rados.rados.0.smithi055.stdout:110: finishing write tid 4 to smithi05583006-49 2023-11-28T14:32:18.140 INFO:tasks.rados.rados.0.smithi055.stdout:110: finishing write tid 5 to smithi05583006-49 2023-11-28T14:32:18.140 INFO:tasks.rados.rados.0.smithi055.stdout:110: oid 49 updating version 0 to 24 2023-11-28T14:32:18.140 INFO:tasks.rados.rados.0.smithi055.stdout:110: oid 49 updating version 24 to 26 2023-11-28T14:32:18.140 INFO:tasks.rados.rados.0.smithi055.stdout:110: oid 49 version 26 is already newer than 23 2023-11-28T14:32:18.140 INFO:tasks.rados.rados.0.smithi055.stdout:110: oid 49 version 26 is already newer than 25 2023-11-28T14:32:18.140 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 49 v 26 (ObjNum 79 snap 2 seq_num 79) dirty exists 2023-11-28T14:32:18.140 INFO:tasks.rados.rados.0.smithi055.stdout:110: left oid 49 (ObjNum 79 snap 2 seq_num 79) 2023-11-28T14:32:18.140 INFO:tasks.rados.rados.0.smithi055.stdout:113: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2023-11-28T14:32:18.145 INFO:tasks.rados.rados.0.smithi055.stdout:113: writing smithi05583006-2 from 561367 to 1020066 tid 1 2023-11-28T14:32:18.150 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:18.150 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:18.150 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:18.150 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:18.150 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:18.150 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:18.151 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:18.151 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:18.151 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:18.151 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:18.151 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:18.151 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:18.153 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:18.154 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:18.154 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-11-28T14:32:18.154 INFO:tasks.rados.rados.0.smithi055.stdout:113: writing smithi05583006-2 from 1573512 to 2314871 tid 2 2023-11-28T14:32:18.160 INFO:tasks.rados.rados.0.smithi055.stdout:113: writing smithi05583006-2 from 2941380 to 3361367 tid 3 2023-11-28T14:32:18.161 INFO:tasks.rados.rados.0.smithi055.stdout:110: done (2 left) 2023-11-28T14:32:18.161 INFO:tasks.rados.rados.0.smithi055.stdout:114: delete oid 8 current snap is 2 2023-11-28T14:32:18.165 INFO:tasks.rados.rados.0.smithi055.stdout:112: expect (ObjNum 70 snap 1 seq_num 70) 2023-11-28T14:32:18.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:18.286+0000 7f146eead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:18.294 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:18.293+0000 7f146eead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:18.307 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:18.307 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:18.307 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:18.307 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:18.307 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:18.307 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:18.307 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:18.307 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:18.307 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:18.307 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:18.307 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:18.308 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:18.311 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:18.312 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-11-28T14:32:18.347 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:32:18.382 INFO:tasks.rados.rados.0.smithi055.stdout:113: finishing write tid 1 to smithi05583006-2 2023-11-28T14:32:18.382 INFO:tasks.rados.rados.0.smithi055.stdout:113: finishing write tid 2 to smithi05583006-2 2023-11-28T14:32:18.382 INFO:tasks.rados.rados.0.smithi055.stdout:113: finishing write tid 3 to smithi05583006-2 2023-11-28T14:32:18.382 INFO:tasks.rados.rados.0.smithi055.stdout:113: finishing write tid 4 to smithi05583006-2 2023-11-28T14:32:18.382 INFO:tasks.rados.rados.0.smithi055.stdout:113: finishing write tid 5 to smithi05583006-2 2023-11-28T14:32:18.382 INFO:tasks.rados.rados.0.smithi055.stdout:113: oid 2 updating version 0 to 55 2023-11-28T14:32:18.383 INFO:tasks.rados.rados.0.smithi055.stdout:113: oid 2 updating version 55 to 57 2023-11-28T14:32:18.383 INFO:tasks.rados.rados.0.smithi055.stdout:113: oid 2 version 57 is already newer than 57 2023-11-28T14:32:18.383 INFO:tasks.rados.rados.0.smithi055.stdout:113: oid 2 version 57 is already newer than 54 2023-11-28T14:32:18.383 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 2 v 57 (ObjNum 81 snap 2 seq_num 81) dirty exists 2023-11-28T14:32:18.383 INFO:tasks.rados.rados.0.smithi055.stdout:113: left oid 2 (ObjNum 81 snap 2 seq_num 81) 2023-11-28T14:32:18.383 INFO:tasks.rados.rados.0.smithi055.stdout:112: done (2 left) 2023-11-28T14:32:18.383 INFO:tasks.rados.rados.0.smithi055.stdout:113: done (1 left) 2023-11-28T14:32:18.383 INFO:tasks.rados.rados.0.smithi055.stdout:114: done (0 left) 2023-11-28T14:32:18.383 INFO:tasks.rados.rados.0.smithi055.stdout:115: read oid 30 snap -1 2023-11-28T14:32:18.383 INFO:tasks.rados.rados.0.smithi055.stdout:115: expect deleted 2023-11-28T14:32:18.383 INFO:tasks.rados.rados.0.smithi055.stdout:116: read oid 24 snap -1 2023-11-28T14:32:18.384 INFO:tasks.rados.rados.0.smithi055.stdout:116: expect (ObjNum 23 snap 0 seq_num 23) 2023-11-28T14:32:18.384 INFO:tasks.rados.rados.0.smithi055.stdout:117: delete oid 21 current snap is 2 2023-11-28T14:32:18.393 INFO:tasks.rados.rados.0.smithi055.stdout:116: expect (ObjNum 23 snap 0 seq_num 23) 2023-11-28T14:32:18.467 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:18.464+0000 7ffbe4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:18.470 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:18.469+0000 7ffbe4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:18.470 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:18.464+0000 7f0b6459e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:18.474 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:18.473+0000 7f0b6459e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:18.487 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:18.487 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:18.487 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:18.487 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:18.487 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:18.487 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:18.488 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:18.488 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:18.488 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:18.488 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:18.488 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:18.488 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:18.489 INFO:tasks.rados.rados.0.smithi055.stdout:115: done (2 left) 2023-11-28T14:32:18.489 INFO:tasks.rados.rados.0.smithi055.stdout:116: done (1 left) 2023-11-28T14:32:18.489 INFO:tasks.rados.rados.0.smithi055.stdout:117: done (0 left) 2023-11-28T14:32:18.489 INFO:tasks.rados.rados.0.smithi055.stdout:118: read oid 50 snap -1 2023-11-28T14:32:18.489 INFO:tasks.rados.rados.0.smithi055.stdout:118: expect (ObjNum 49 snap 0 seq_num 49) 2023-11-28T14:32:18.489 INFO:tasks.rados.rados.0.smithi055.stdout:119: snap_create 2023-11-28T14:32:18.490 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:18.490 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:18.491 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-11-28T14:32:18.506 INFO:tasks.rados.rados.0.smithi055.stdout:118: expect (ObjNum 49 snap 0 seq_num 49) 2023-11-28T14:32:18.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:18.614+0000 7ff2d6e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:18.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:18.622+0000 7ff2d6e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:18.635 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:18.635 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:18.635 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:18.635 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:18.635 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:18.635 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:18.635 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:18.636 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:18.636 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:18.636 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:18.636 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:18.636 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:18.638 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:19.200 INFO:tasks.rados.rados.0.smithi055.stdout:118: done (1 left) 2023-11-28T14:32:19.200 INFO:tasks.rados.rados.0.smithi055.stdout:119: done (0 left) 2023-11-28T14:32:19.200 INFO:tasks.rados.rados.0.smithi055.stdout:120: snap_create 2023-11-28T14:32:20.197 INFO:tasks.rados.rados.0.smithi055.stdout:120: done (0 left) 2023-11-28T14:32:20.198 INFO:tasks.rados.rados.0.smithi055.stdout:121: write oid 42 current snap is 4 2023-11-28T14:32:20.198 INFO:tasks.rados.rados.0.smithi055.stdout:121: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2023-11-28T14:32:20.203 INFO:tasks.rados.rados.0.smithi055.stdout:121: writing smithi05583006-42 from 611788 to 1012587 tid 1 2023-11-28T14:32:20.211 INFO:tasks.rados.rados.0.smithi055.stdout:121: writing smithi05583006-42 from 1748958 to 2237644 tid 2 2023-11-28T14:32:20.219 INFO:tasks.rados.rados.0.smithi055.stdout:121: writing smithi05583006-42 from 2907296 to 3411788 tid 3 2023-11-28T14:32:20.220 INFO:tasks.rados.rados.0.smithi055.stdout:122: snap_remove snap 3 2023-11-28T14:32:20.544 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:32:20.659 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:20.657+0000 7fe8e4755640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:20.662 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:20.661+0000 7fe8e4755640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:21.220 INFO:tasks.rados.rados.0.smithi055.stdout:122: done (1 left) 2023-11-28T14:32:21.220 INFO:tasks.rados.rados.0.smithi055.stdout:123: snap_remove snap 2 2023-11-28T14:32:22.218 INFO:tasks.rados.rados.0.smithi055.stdout:123: done (1 left) 2023-11-28T14:32:22.218 INFO:tasks.rados.rados.0.smithi055.stdout:124: write (excl) oid 41 current snap is 4 2023-11-28T14:32:22.218 INFO:tasks.rados.rados.0.smithi055.stdout:121: finishing write tid 1 to smithi05583006-42 2023-11-28T14:32:22.218 INFO:tasks.rados.rados.0.smithi055.stdout:124: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2023-11-28T14:32:22.223 INFO:tasks.rados.rados.0.smithi055.stdout:124: writing smithi05583006-41 from 628595 to 1104643 tid 1 2023-11-28T14:32:22.233 INFO:tasks.rados.rados.0.smithi055.stdout:124: writing smithi05583006-41 from 1807440 to 2439784 tid 2 2023-11-28T14:32:22.240 INFO:tasks.rados.rados.0.smithi055.stdout:124: writing smithi05583006-41 from 2934719 to 3387193 tid 3 2023-11-28T14:32:22.240 INFO:tasks.rados.rados.0.smithi055.stdout:125: write oid 35 current snap is 4 2023-11-28T14:32:22.240 INFO:tasks.rados.rados.0.smithi055.stdout:125: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2023-11-28T14:32:22.248 INFO:tasks.rados.rados.0.smithi055.stdout:125: writing smithi05583006-35 from 645402 to 1196699 tid 1 2023-11-28T14:32:22.255 INFO:tasks.rados.rados.0.smithi055.stdout:125: writing smithi05583006-35 from 1749569 to 2241924 tid 2 2023-11-28T14:32:22.262 INFO:tasks.rados.rados.0.smithi055.stdout:125: writing smithi05583006-35 from 2845789 to 3445402 tid 3 2023-11-28T14:32:22.262 INFO:tasks.rados.rados.0.smithi055.stdout:126: snap_create 2023-11-28T14:32:22.263 INFO:tasks.rados.rados.0.smithi055.stdout:121: finishing write tid 2 to smithi05583006-42 2023-11-28T14:32:22.263 INFO:tasks.rados.rados.0.smithi055.stdout:121: finishing write tid 3 to smithi05583006-42 2023-11-28T14:32:22.263 INFO:tasks.rados.rados.0.smithi055.stdout:121: finishing write tid 4 to smithi05583006-42 2023-11-28T14:32:22.263 INFO:tasks.rados.rados.0.smithi055.stdout:121: finishing write tid 5 to smithi05583006-42 2023-11-28T14:32:22.263 INFO:tasks.rados.rados.0.smithi055.stdout:121: oid 42 updating version 0 to 16 2023-11-28T14:32:22.263 INFO:tasks.rados.rados.0.smithi055.stdout:121: oid 42 updating version 16 to 17 2023-11-28T14:32:22.263 INFO:tasks.rados.rados.0.smithi055.stdout:121: oid 42 version 17 is already newer than 14 2023-11-28T14:32:22.263 INFO:tasks.rados.rados.0.smithi055.stdout:121: oid 42 version 17 is already newer than 16 2023-11-28T14:32:22.263 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 42 v 17 (ObjNum 84 snap 4 seq_num 84) dirty exists 2023-11-28T14:32:22.263 INFO:tasks.rados.rados.0.smithi055.stdout:121: left oid 42 (ObjNum 84 snap 4 seq_num 84) 2023-11-28T14:32:22.331 INFO:tasks.rados.rados.0.smithi055.stdout:124: finishing write tid 1 to smithi05583006-41 2023-11-28T14:32:22.333 INFO:tasks.rados.rados.0.smithi055.stdout:125: finishing write tid 1 to smithi05583006-35 2023-11-28T14:32:22.358 INFO:tasks.rados.rados.0.smithi055.stdout:125: finishing write tid 2 to smithi05583006-35 2023-11-28T14:32:22.358 INFO:tasks.rados.rados.0.smithi055.stdout:124: finishing write tid 2 to smithi05583006-41 2023-11-28T14:32:22.380 INFO:tasks.rados.rados.0.smithi055.stdout:124: finishing write tid 3 to smithi05583006-41 2023-11-28T14:32:22.381 INFO:tasks.rados.rados.0.smithi055.stdout:125: finishing write tid 3 to smithi05583006-35 2023-11-28T14:32:22.391 INFO:tasks.rados.rados.0.smithi055.stdout:124: finishing write tid 4 to smithi05583006-41 2023-11-28T14:32:22.391 INFO:tasks.rados.rados.0.smithi055.stdout:125: finishing write tid 4 to smithi05583006-35 2023-11-28T14:32:22.393 INFO:tasks.rados.rados.0.smithi055.stdout:124: finishing write tid 5 to smithi05583006-41 2023-11-28T14:32:22.393 INFO:tasks.rados.rados.0.smithi055.stdout:124: oid 41 updating version 0 to 59 2023-11-28T14:32:22.393 INFO:tasks.rados.rados.0.smithi055.stdout:124: oid 41 updating version 59 to 61 2023-11-28T14:32:22.393 INFO:tasks.rados.rados.0.smithi055.stdout:124: oid 41 version 61 is already newer than 60 2023-11-28T14:32:22.393 INFO:tasks.rados.rados.0.smithi055.stdout:124: oid 41 version 61 is already newer than 58 2023-11-28T14:32:22.393 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 41 v 61 (ObjNum 85 snap 4 seq_num 85) dirty exists 2023-11-28T14:32:22.393 INFO:tasks.rados.rados.0.smithi055.stdout:124: left oid 41 (ObjNum 85 snap 4 seq_num 85) 2023-11-28T14:32:22.393 INFO:tasks.rados.rados.0.smithi055.stdout:125: finishing write tid 5 to smithi05583006-35 2023-11-28T14:32:22.393 INFO:tasks.rados.rados.0.smithi055.stdout:125: oid 35 updating version 0 to 20 2023-11-28T14:32:22.393 INFO:tasks.rados.rados.0.smithi055.stdout:125: oid 35 updating version 20 to 23 2023-11-28T14:32:22.394 INFO:tasks.rados.rados.0.smithi055.stdout:125: oid 35 version 23 is already newer than 22 2023-11-28T14:32:22.394 INFO:tasks.rados.rados.0.smithi055.stdout:125: oid 35 version 23 is already newer than 21 2023-11-28T14:32:22.394 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 35 v 23 (ObjNum 86 snap 4 seq_num 86) dirty exists 2023-11-28T14:32:22.394 INFO:tasks.rados.rados.0.smithi055.stdout:125: left oid 35 (ObjNum 86 snap 4 seq_num 86) 2023-11-28T14:32:22.733 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:32:22.851 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:22.849+0000 7ff72ad1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:22.854 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:22.853+0000 7ff72ad1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:23.238 INFO:tasks.rados.rados.0.smithi055.stdout:121: done (3 left) 2023-11-28T14:32:23.238 INFO:tasks.rados.rados.0.smithi055.stdout:124: done (2 left) 2023-11-28T14:32:23.238 INFO:tasks.rados.rados.0.smithi055.stdout:125: done (1 left) 2023-11-28T14:32:23.238 INFO:tasks.rados.rados.0.smithi055.stdout:126: done (0 left) 2023-11-28T14:32:23.238 INFO:tasks.rados.rados.0.smithi055.stdout:127: write (excl) oid 42 current snap is 5 2023-11-28T14:32:23.238 INFO:tasks.rados.rados.0.smithi055.stdout:127: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2023-11-28T14:32:23.247 INFO:tasks.rados.rados.0.smithi055.stdout:127: writing smithi05583006-42 from 662209 to 1288755 tid 1 2023-11-28T14:32:23.255 INFO:tasks.rados.rados.0.smithi055.stdout:127: writing smithi05583006-42 from 1808051 to 2444064 tid 2 2023-11-28T14:32:23.261 INFO:tasks.rados.rados.0.smithi055.stdout:127: writing smithi05583006-42 from 2873212 to 3462209 tid 3 2023-11-28T14:32:23.261 INFO:tasks.rados.rados.0.smithi055.stdout:128: snap_remove snap 4 2023-11-28T14:32:23.639 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:32:23.639 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-11-28T14:32:23.759 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:23.750+0000 7f6637b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:23.760 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:23.759+0000 7f6637b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:24.237 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.0 to 1 (10000) 2023-11-28T14:32:24.244 INFO:tasks.rados.rados.0.smithi055.stdout:128: done (1 left) 2023-11-28T14:32:24.245 INFO:tasks.rados.rados.0.smithi055.stdout:129: read oid 2 snap -1 2023-11-28T14:32:24.245 INFO:tasks.rados.rados.0.smithi055.stdout:129: expect (ObjNum 81 snap 2 seq_num 81) 2023-11-28T14:32:24.245 INFO:tasks.rados.rados.0.smithi055.stdout:130: snap_create 2023-11-28T14:32:24.245 INFO:tasks.rados.rados.0.smithi055.stdout:127: finishing write tid 1 to smithi05583006-42 2023-11-28T14:32:24.245 INFO:tasks.rados.rados.0.smithi055.stdout:127: finishing write tid 2 to smithi05583006-42 2023-11-28T14:32:24.245 INFO:tasks.rados.rados.0.smithi055.stdout:127: finishing write tid 3 to smithi05583006-42 2023-11-28T14:32:24.245 INFO:tasks.rados.rados.0.smithi055.stdout:127: finishing write tid 4 to smithi05583006-42 2023-11-28T14:32:24.245 INFO:tasks.rados.rados.0.smithi055.stdout:127: finishing write tid 5 to smithi05583006-42 2023-11-28T14:32:24.245 INFO:tasks.rados.rados.0.smithi055.stdout:127: oid 42 updating version 0 to 22 2023-11-28T14:32:24.245 INFO:tasks.rados.rados.0.smithi055.stdout:127: oid 42 updating version 22 to 24 2023-11-28T14:32:24.245 INFO:tasks.rados.rados.0.smithi055.stdout:127: oid 42 version 24 is already newer than 24 2023-11-28T14:32:24.245 INFO:tasks.rados.rados.0.smithi055.stdout:127: oid 42 version 24 is already newer than 21 2023-11-28T14:32:24.246 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 42 v 24 (ObjNum 87 snap 5 seq_num 87) dirty exists 2023-11-28T14:32:24.246 INFO:tasks.rados.rados.0.smithi055.stdout:127: left oid 42 (ObjNum 87 snap 5 seq_num 87) 2023-11-28T14:32:24.250 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-11-28T14:32:24.343 INFO:tasks.rados.rados.0.smithi055.stdout:129: expect (ObjNum 81 snap 2 seq_num 81) 2023-11-28T14:32:24.375 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:24.368+0000 7f495659e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:24.377 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:24.377+0000 7f495659e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:24.925 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:32:25.040 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:25.037+0000 7f6f4c0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:25.043 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:25.042+0000 7f6f4c0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:25.242 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.1 to 1 (10000) 2023-11-28T14:32:25.248 INFO:tasks.rados.rados.0.smithi055.stdout:127: done (2 left) 2023-11-28T14:32:25.248 INFO:tasks.rados.rados.0.smithi055.stdout:129: done (1 left) 2023-11-28T14:32:25.248 INFO:tasks.rados.rados.0.smithi055.stdout:130: done (0 left) 2023-11-28T14:32:25.248 INFO:tasks.rados.rados.0.smithi055.stdout:131: snap_create 2023-11-28T14:32:25.256 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-11-28T14:32:25.386 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:25.380+0000 7f386695e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:25.389 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:25.388+0000 7f386695e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:26.243 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.2 to 1 (10000) 2023-11-28T14:32:26.248 INFO:tasks.rados.rados.0.smithi055.stdout:131: done (0 left) 2023-11-28T14:32:26.248 INFO:tasks.rados.rados.0.smithi055.stdout:132: snap_create 2023-11-28T14:32:26.259 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-11-28T14:32:26.380 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:26.373+0000 7ff4fa19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:26.383 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:26.382+0000 7ff4fa19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:27.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:32:27.228 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:27.226+0000 7faa9da6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:27.231 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:27.230+0000 7faa9da6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:27.249 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.3 to 1 (10000) 2023-11-28T14:32:27.254 INFO:tasks.rados.rados.0.smithi055.stdout:132: done (0 left) 2023-11-28T14:32:27.254 INFO:tasks.rados.rados.0.smithi055.stdout:133: snap_remove snap 7 2023-11-28T14:32:27.265 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-11-28T14:32:27.265 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:32:27.383 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:27.378+0000 7f073395e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:27.386 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:27.385+0000 7f073395e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:27.571 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:27.571 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:32:27.582 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2023-11-28T14:32:27.251420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114658091,"num_objects":62,"num_object_clones":17,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":344,"num_read_kb":114297,"num_write":467,"num_write_kb":94687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":407,"ondisk_log_size":407,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":290976,"kb_used_data":182984,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377196384,"statfs":{"total":386547056640,"available":386249097216,"internally_reserved":0,"allocated":187375616,"data_stored":188042236,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":554631,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020061"},"pg_stats":[{"pgid":"3.f","version":"27'61","reported_seq":96,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.381814+0000","last_change":"2023-11-28T14:32:22.254726+0000","last_active":"2023-11-28T14:32:22.381814+0000","last_peered":"2023-11-28T14:32:22.381814+0000","last_clean":"2023-11-28T14:32:22.381814+0000","last_became_active":"2023-11-28T14:32:09.321992+0000","last_became_peered":"2023-11-28T14:32:09.321992+0000","last_unstale":"2023-11-28T14:32:22.381814+0000","last_undegraded":"2023-11-28T14:32:22.381814+0000","last_fullsized":"2023-11-28T14:32:22.381814+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15159749,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":39,"num_read_kb":14127,"num_write":61,"num_write_kb":17181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"19'3","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T14:32:21.230627+0000","last_change":"2023-11-28T14:32:21.234553+0000","last_active":"2023-11-28T14:32:21.230627+0000","last_peered":"2023-11-28T14:32:21.230627+0000","last_clean":"2023-11-28T14:32:21.230627+0000","last_became_active":"2023-11-28T14:32:09.308848+0000","last_became_peered":"2023-11-28T14:32:09.308848+0000","last_unstale":"2023-11-28T14:32:21.230627+0000","last_undegraded":"2023-11-28T14:32:21.230627+0000","last_fullsized":"2023-11-28T14:32:21.230627+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":6433,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'15","reported_seq":52,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.501317+0000","last_change":"2023-11-28T14:32:17.513558+0000","last_active":"2023-11-28T14:32:20.501317+0000","last_peered":"2023-11-28T14:32:20.501317+0000","last_clean":"2023-11-28T14:32:20.501317+0000","last_became_active":"2023-11-28T14:32:09.337560+0000","last_became_peered":"2023-11-28T14:32:09.337560+0000","last_unstale":"2023-11-28T14:32:20.501317+0000","last_undegraded":"2023-11-28T14:32:20.501317+0000","last_fullsized":"2023-11-28T14:32:20.501317+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7848770,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":24013,"num_write":16,"num_write_kb":4334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"25'17","reported_seq":49,"reported_epoch":27,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:32:22.245495+0000","last_change":"2023-11-28T14:32:22.245495+0000","last_active":"2023-11-28T14:32:22.245495+0000","last_peered":"2023-11-28T14:32:22.245495+0000","last_clean":"2023-11-28T14:32:22.245495+0000","last_became_active":"2023-11-28T14:32:09.344782+0000","last_became_peered":"2023-11-28T14:32:09.344782+0000","last_unstale":"2023-11-28T14:32:22.245495+0000","last_undegraded":"2023-11-28T14:32:22.245495+0000","last_fullsized":"2023-11-28T14:32:22.245495+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11008453,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":9911,"num_write":21,"num_write_kb":6067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'23","reported_seq":58,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.382078+0000","last_change":"2023-11-28T14:32:22.351600+0000","last_active":"2023-11-28T14:32:22.382078+0000","last_peered":"2023-11-28T14:32:22.382078+0000","last_clean":"2023-11-28T14:32:22.382078+0000","last_became_active":"2023-11-28T14:32:09.326269+0000","last_became_peered":"2023-11-28T14:32:09.326269+0000","last_unstale":"2023-11-28T14:32:22.382078+0000","last_undegraded":"2023-11-28T14:32:22.382078+0000","last_fullsized":"2023-11-28T14:32:22.382078+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8470629,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":7024,"num_write":26,"num_write_kb":7179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"18'9","reported_seq":41,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.250686+0000","last_change":"2023-11-28T14:32:22.272291+0000","last_active":"2023-11-28T14:32:22.250686+0000","last_peered":"2023-11-28T14:32:22.250686+0000","last_clean":"2023-11-28T14:32:22.250686+0000","last_became_active":"2023-11-28T14:32:09.308846+0000","last_became_peered":"2023-11-28T14:32:09.308846+0000","last_unstale":"2023-11-28T14:32:22.250686+0000","last_undegraded":"2023-11-28T14:32:22.250686+0000","last_fullsized":"2023-11-28T14:32:22.250686+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6991613,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5566,"num_write":12,"num_write_kb":3092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'21","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.253216+0000","last_change":"2023-11-28T14:32:22.256505+0000","last_active":"2023-11-28T14:32:22.253216+0000","last_peered":"2023-11-28T14:32:22.253216+0000","last_clean":"2023-11-28T14:32:22.253216+0000","last_became_active":"2023-11-28T14:32:09.316660+0000","last_became_peered":"2023-11-28T14:32:09.316660+0000","last_unstale":"2023-11-28T14:32:22.253216+0000","last_undegraded":"2023-11-28T14:32:22.253216+0000","last_fullsized":"2023-11-28T14:32:22.253216+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6504243,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":3445,"num_write":23,"num_write_kb":6709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'13","reported_seq":44,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.502712+0000","last_change":"2023-11-28T14:32:17.342305+0000","last_active":"2023-11-28T14:32:20.502712+0000","last_peered":"2023-11-28T14:32:20.502712+0000","last_clean":"2023-11-28T14:32:20.502712+0000","last_became_active":"2023-11-28T14:32:09.358025+0000","last_became_peered":"2023-11-28T14:32:09.358025+0000","last_unstale":"2023-11-28T14:32:20.502712+0000","last_undegraded":"2023-11-28T14:32:20.502712+0000","last_fullsized":"2023-11-28T14:32:20.502712+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7075648,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":3594,"num_write":16,"num_write_kb":3963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":164,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.278277+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:32:22.278277+0000","last_peered":"2023-11-28T14:32:22.278277+0000","last_clean":"2023-11-28T14:32:22.278277+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:32:22.278277+0000","last_undegraded":"2023-11-28T14:32:22.278277+0000","last_fullsized":"2023-11-28T14:32:22.278277+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.506590+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:32:20.506590+0000","last_peered":"2023-11-28T14:32:20.506590+0000","last_clean":"2023-11-28T14:32:20.506590+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:32:20.506590+0000","last_undegraded":"2023-11-28T14:32:20.506590+0000","last_fullsized":"2023-11-28T14:32:20.506590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"23'60","reported_seq":100,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.501953+0000","last_change":"2023-11-28T14:32:17.670263+0000","last_active":"2023-11-28T14:32:20.501953+0000","last_peered":"2023-11-28T14:32:20.501953+0000","last_clean":"2023-11-28T14:32:20.501953+0000","last_became_active":"2023-11-28T14:32:09.335805+0000","last_became_peered":"2023-11-28T14:32:09.335805+0000","last_unstale":"2023-11-28T14:32:20.501953+0000","last_undegraded":"2023-11-28T14:32:20.501953+0000","last_fullsized":"2023-11-28T14:32:20.501953+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19226977,"num_objects":11,"num_object_clones":4,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":50,"num_read_kb":23833,"num_write":60,"num_write_kb":16715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.246497+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:32:22.246497+0000","last_peered":"2023-11-28T14:32:22.246497+0000","last_clean":"2023-11-28T14:32:22.246497+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:32:22.246497+0000","last_undegraded":"2023-11-28T14:32:22.246497+0000","last_fullsized":"2023-11-28T14:32:22.246497+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"18'13","reported_seq":46,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.287116+0000","last_change":"2023-11-28T14:32:22.289940+0000","last_active":"2023-11-28T14:32:22.287116+0000","last_peered":"2023-11-28T14:32:22.287116+0000","last_clean":"2023-11-28T14:32:22.287116+0000","last_became_active":"2023-11-28T14:32:09.350412+0000","last_became_peered":"2023-11-28T14:32:09.350412+0000","last_unstale":"2023-11-28T14:32:22.287116+0000","last_undegraded":"2023-11-28T14:32:22.287116+0000","last_fullsized":"2023-11-28T14:32:22.287116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7680700,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":16,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.502715+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:32:20.502715+0000","last_peered":"2023-11-28T14:32:20.502715+0000","last_clean":"2023-11-28T14:32:20.502715+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:32:20.502715+0000","last_undegraded":"2023-11-28T14:32:20.502715+0000","last_fullsized":"2023-11-28T14:32:20.502715+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'10","reported_seq":38,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.281524+0000","last_change":"2023-11-28T14:32:22.285182+0000","last_active":"2023-11-28T14:32:22.281524+0000","last_peered":"2023-11-28T14:32:22.281524+0000","last_clean":"2023-11-28T14:32:22.281524+0000","last_became_active":"2023-11-28T14:32:09.343664+0000","last_became_peered":"2023-11-28T14:32:09.343664+0000","last_unstale":"2023-11-28T14:32:22.281524+0000","last_undegraded":"2023-11-28T14:32:22.281524+0000","last_fullsized":"2023-11-28T14:32:22.281524+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941192,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":2187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.245219+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:32:22.245219+0000","last_peered":"2023-11-28T14:32:22.245219+0000","last_clean":"2023-11-28T14:32:22.245219+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:32:22.245219+0000","last_undegraded":"2023-11-28T14:32:22.245219+0000","last_fullsized":"2023-11-28T14:32:22.245219+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"19'3","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T14:32:21.225477+0000","last_change":"2023-11-28T14:32:21.235288+0000","last_active":"2023-11-28T14:32:21.225477+0000","last_peered":"2023-11-28T14:32:21.225477+0000","last_clean":"2023-11-28T14:32:21.225477+0000","last_became_active":"2023-11-28T14:32:09.311653+0000","last_became_peered":"2023-11-28T14:32:09.311653+0000","last_unstale":"2023-11-28T14:32:21.225477+0000","last_undegraded":"2023-11-28T14:32:21.225477+0000","last_fullsized":"2023-11-28T14:32:21.225477+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016807,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"23'16","reported_seq":40,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.510117+0000","last_change":"2023-11-28T14:32:17.473937+0000","last_active":"2023-11-28T14:32:20.510117+0000","last_peered":"2023-11-28T14:32:20.510117+0000","last_clean":"2023-11-28T14:32:20.510117+0000","last_became_active":"2023-11-28T14:32:09.316944+0000","last_became_peered":"2023-11-28T14:32:09.316944+0000","last_unstale":"2023-11-28T14:32:20.510117+0000","last_undegraded":"2023-11-28T14:32:20.510117+0000","last_fullsized":"2023-11-28T14:32:20.510117+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3193297,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":16,"num_write_kb":4613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T14:32:21.228743+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:32:21.228743+0000","last_peered":"2023-11-28T14:32:21.228743+0000","last_clean":"2023-11-28T14:32:21.228743+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:32:21.228743+0000","last_undegraded":"2023-11-28T14:32:21.228743+0000","last_fullsized":"2023-11-28T14:32:21.228743+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"23'13","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T14:32:21.227937+0000","last_change":"2023-11-28T14:32:21.234090+0000","last_active":"2023-11-28T14:32:21.227937+0000","last_peered":"2023-11-28T14:32:21.227937+0000","last_clean":"2023-11-28T14:32:21.227937+0000","last_became_active":"2023-11-28T14:32:09.309510+0000","last_became_peered":"2023-11-28T14:32:09.309510+0000","last_unstale":"2023-11-28T14:32:21.227937+0000","last_undegraded":"2023-11-28T14:32:21.227937+0000","last_fullsized":"2023-11-28T14:32:21.227937+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2386561,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3260,"num_write":12,"num_write_kb":3313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.284201+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:32:22.284201+0000","last_peered":"2023-11-28T14:32:22.284201+0000","last_clean":"2023-11-28T14:32:22.284201+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:32:22.284201+0000","last_undegraded":"2023-11-28T14:32:22.284201+0000","last_fullsized":"2023-11-28T14:32:22.284201+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"23'27","reported_seq":58,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.304334+0000","last_change":"2023-11-28T14:32:22.325131+0000","last_active":"2023-11-28T14:32:22.304334+0000","last_peered":"2023-11-28T14:32:22.304334+0000","last_clean":"2023-11-28T14:32:22.304334+0000","last_became_active":"2023-11-28T14:32:09.356991+0000","last_became_peered":"2023-11-28T14:32:09.356991+0000","last_unstale":"2023-11-28T14:32:22.304334+0000","last_undegraded":"2023-11-28T14:32:22.304334+0000","last_fullsized":"2023-11-28T14:32:22.304334+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151296,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":3102,"num_write":26,"num_write_kb":8152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T14:32:21.225295+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:32:21.225295+0000","last_peered":"2023-11-28T14:32:21.225295+0000","last_clean":"2023-11-28T14:32:21.225295+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:32:21.225295+0000","last_undegraded":"2023-11-28T14:32:21.225295+0000","last_fullsized":"2023-11-28T14:32:21.225295+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"19'6","reported_seq":42,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T14:32:21.229594+0000","last_change":"2023-11-28T14:32:21.235134+0000","last_active":"2023-11-28T14:32:21.229594+0000","last_peered":"2023-11-28T14:32:21.229594+0000","last_clean":"2023-11-28T14:32:21.229594+0000","last_became_active":"2023-11-28T14:32:09.337548+0000","last_became_peered":"2023-11-28T14:32:09.337548+0000","last_unstale":"2023-11-28T14:32:21.229594+0000","last_undegraded":"2023-11-28T14:32:21.229594+0000","last_fullsized":"2023-11-28T14:32:21.229594+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":9887,"num_write":8,"num_write_kb":2059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.503533+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:32:20.503533+0000","last_peered":"2023-11-28T14:32:20.503533+0000","last_clean":"2023-11-28T14:32:20.503533+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:32:20.503533+0000","last_undegraded":"2023-11-28T14:32:20.503533+0000","last_fullsized":"2023-11-28T14:32:20.503533+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":114067723,"num_objects":60,"num_object_clones":17,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":241,"num_read_kb":114206,"num_write":330,"num_write_kb":92793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":310,"ondisk_log_size":310,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75944,"kb_used_data":49120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94295896,"statfs":{"total":96636764160,"available":96558997504,"internally_reserved":0,"allocated":50298880,"data_stored":50793812,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705690,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81412,"kb_used_data":54588,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94290428,"statfs":{"total":96636764160,"available":96553398272,"internally_reserved":0,"allocated":55898112,"data_stored":56111818,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":58248,"kb_used_data":31424,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94313592,"statfs":{"total":96636764160,"available":96577118208,"internally_reserved":0,"allocated":32178176,"data_stored":32599026,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705690,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75372,"kb_used_data":47852,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94296468,"statfs":{"total":96636764160,"available":96559583232,"internally_reserved":0,"allocated":49000448,"data_stored":48537580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:32:27.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:32:27.696 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:27.691+0000 7f99ef80f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:27.699 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:27.698+0000 7f99ef80f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:27.883 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:27.883 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:32:27.895 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2023-11-28T14:32:27.251420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114658091,"num_objects":62,"num_object_clones":17,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":344,"num_read_kb":114297,"num_write":467,"num_write_kb":94687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":407,"ondisk_log_size":407,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":290976,"kb_used_data":182984,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377196384,"statfs":{"total":386547056640,"available":386249097216,"internally_reserved":0,"allocated":187375616,"data_stored":188042236,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":554631,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020061"},"pg_stats":[{"pgid":"3.f","version":"27'61","reported_seq":96,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.381814+0000","last_change":"2023-11-28T14:32:22.254726+0000","last_active":"2023-11-28T14:32:22.381814+0000","last_peered":"2023-11-28T14:32:22.381814+0000","last_clean":"2023-11-28T14:32:22.381814+0000","last_became_active":"2023-11-28T14:32:09.321992+0000","last_became_peered":"2023-11-28T14:32:09.321992+0000","last_unstale":"2023-11-28T14:32:22.381814+0000","last_undegraded":"2023-11-28T14:32:22.381814+0000","last_fullsized":"2023-11-28T14:32:22.381814+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15159749,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":39,"num_read_kb":14127,"num_write":61,"num_write_kb":17181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"19'3","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T14:32:21.230627+0000","last_change":"2023-11-28T14:32:21.234553+0000","last_active":"2023-11-28T14:32:21.230627+0000","last_peered":"2023-11-28T14:32:21.230627+0000","last_clean":"2023-11-28T14:32:21.230627+0000","last_became_active":"2023-11-28T14:32:09.308848+0000","last_became_peered":"2023-11-28T14:32:09.308848+0000","last_unstale":"2023-11-28T14:32:21.230627+0000","last_undegraded":"2023-11-28T14:32:21.230627+0000","last_fullsized":"2023-11-28T14:32:21.230627+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":6433,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'15","reported_seq":52,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.501317+0000","last_change":"2023-11-28T14:32:17.513558+0000","last_active":"2023-11-28T14:32:20.501317+0000","last_peered":"2023-11-28T14:32:20.501317+0000","last_clean":"2023-11-28T14:32:20.501317+0000","last_became_active":"2023-11-28T14:32:09.337560+0000","last_became_peered":"2023-11-28T14:32:09.337560+0000","last_unstale":"2023-11-28T14:32:20.501317+0000","last_undegraded":"2023-11-28T14:32:20.501317+0000","last_fullsized":"2023-11-28T14:32:20.501317+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7848770,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":24013,"num_write":16,"num_write_kb":4334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"25'17","reported_seq":49,"reported_epoch":27,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:32:22.245495+0000","last_change":"2023-11-28T14:32:22.245495+0000","last_active":"2023-11-28T14:32:22.245495+0000","last_peered":"2023-11-28T14:32:22.245495+0000","last_clean":"2023-11-28T14:32:22.245495+0000","last_became_active":"2023-11-28T14:32:09.344782+0000","last_became_peered":"2023-11-28T14:32:09.344782+0000","last_unstale":"2023-11-28T14:32:22.245495+0000","last_undegraded":"2023-11-28T14:32:22.245495+0000","last_fullsized":"2023-11-28T14:32:22.245495+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11008453,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":9911,"num_write":21,"num_write_kb":6067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'23","reported_seq":58,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.382078+0000","last_change":"2023-11-28T14:32:22.351600+0000","last_active":"2023-11-28T14:32:22.382078+0000","last_peered":"2023-11-28T14:32:22.382078+0000","last_clean":"2023-11-28T14:32:22.382078+0000","last_became_active":"2023-11-28T14:32:09.326269+0000","last_became_peered":"2023-11-28T14:32:09.326269+0000","last_unstale":"2023-11-28T14:32:22.382078+0000","last_undegraded":"2023-11-28T14:32:22.382078+0000","last_fullsized":"2023-11-28T14:32:22.382078+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8470629,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":7024,"num_write":26,"num_write_kb":7179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"18'9","reported_seq":41,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.250686+0000","last_change":"2023-11-28T14:32:22.272291+0000","last_active":"2023-11-28T14:32:22.250686+0000","last_peered":"2023-11-28T14:32:22.250686+0000","last_clean":"2023-11-28T14:32:22.250686+0000","last_became_active":"2023-11-28T14:32:09.308846+0000","last_became_peered":"2023-11-28T14:32:09.308846+0000","last_unstale":"2023-11-28T14:32:22.250686+0000","last_undegraded":"2023-11-28T14:32:22.250686+0000","last_fullsized":"2023-11-28T14:32:22.250686+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6991613,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5566,"num_write":12,"num_write_kb":3092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'21","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.253216+0000","last_change":"2023-11-28T14:32:22.256505+0000","last_active":"2023-11-28T14:32:22.253216+0000","last_peered":"2023-11-28T14:32:22.253216+0000","last_clean":"2023-11-28T14:32:22.253216+0000","last_became_active":"2023-11-28T14:32:09.316660+0000","last_became_peered":"2023-11-28T14:32:09.316660+0000","last_unstale":"2023-11-28T14:32:22.253216+0000","last_undegraded":"2023-11-28T14:32:22.253216+0000","last_fullsized":"2023-11-28T14:32:22.253216+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6504243,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":3445,"num_write":23,"num_write_kb":6709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'13","reported_seq":44,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.502712+0000","last_change":"2023-11-28T14:32:17.342305+0000","last_active":"2023-11-28T14:32:20.502712+0000","last_peered":"2023-11-28T14:32:20.502712+0000","last_clean":"2023-11-28T14:32:20.502712+0000","last_became_active":"2023-11-28T14:32:09.358025+0000","last_became_peered":"2023-11-28T14:32:09.358025+0000","last_unstale":"2023-11-28T14:32:20.502712+0000","last_undegraded":"2023-11-28T14:32:20.502712+0000","last_fullsized":"2023-11-28T14:32:20.502712+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7075648,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":3594,"num_write":16,"num_write_kb":3963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":164,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.278277+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:32:22.278277+0000","last_peered":"2023-11-28T14:32:22.278277+0000","last_clean":"2023-11-28T14:32:22.278277+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:32:22.278277+0000","last_undegraded":"2023-11-28T14:32:22.278277+0000","last_fullsized":"2023-11-28T14:32:22.278277+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.506590+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:32:20.506590+0000","last_peered":"2023-11-28T14:32:20.506590+0000","last_clean":"2023-11-28T14:32:20.506590+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:32:20.506590+0000","last_undegraded":"2023-11-28T14:32:20.506590+0000","last_fullsized":"2023-11-28T14:32:20.506590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"23'60","reported_seq":100,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.501953+0000","last_change":"2023-11-28T14:32:17.670263+0000","last_active":"2023-11-28T14:32:20.501953+0000","last_peered":"2023-11-28T14:32:20.501953+0000","last_clean":"2023-11-28T14:32:20.501953+0000","last_became_active":"2023-11-28T14:32:09.335805+0000","last_became_peered":"2023-11-28T14:32:09.335805+0000","last_unstale":"2023-11-28T14:32:20.501953+0000","last_undegraded":"2023-11-28T14:32:20.501953+0000","last_fullsized":"2023-11-28T14:32:20.501953+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19226977,"num_objects":11,"num_object_clones":4,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":50,"num_read_kb":23833,"num_write":60,"num_write_kb":16715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.246497+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:32:22.246497+0000","last_peered":"2023-11-28T14:32:22.246497+0000","last_clean":"2023-11-28T14:32:22.246497+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:32:22.246497+0000","last_undegraded":"2023-11-28T14:32:22.246497+0000","last_fullsized":"2023-11-28T14:32:22.246497+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"18'13","reported_seq":46,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.287116+0000","last_change":"2023-11-28T14:32:22.289940+0000","last_active":"2023-11-28T14:32:22.287116+0000","last_peered":"2023-11-28T14:32:22.287116+0000","last_clean":"2023-11-28T14:32:22.287116+0000","last_became_active":"2023-11-28T14:32:09.350412+0000","last_became_peered":"2023-11-28T14:32:09.350412+0000","last_unstale":"2023-11-28T14:32:22.287116+0000","last_undegraded":"2023-11-28T14:32:22.287116+0000","last_fullsized":"2023-11-28T14:32:22.287116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7680700,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":16,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.502715+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:32:20.502715+0000","last_peered":"2023-11-28T14:32:20.502715+0000","last_clean":"2023-11-28T14:32:20.502715+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:32:20.502715+0000","last_undegraded":"2023-11-28T14:32:20.502715+0000","last_fullsized":"2023-11-28T14:32:20.502715+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'10","reported_seq":38,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.281524+0000","last_change":"2023-11-28T14:32:22.285182+0000","last_active":"2023-11-28T14:32:22.281524+0000","last_peered":"2023-11-28T14:32:22.281524+0000","last_clean":"2023-11-28T14:32:22.281524+0000","last_became_active":"2023-11-28T14:32:09.343664+0000","last_became_peered":"2023-11-28T14:32:09.343664+0000","last_unstale":"2023-11-28T14:32:22.281524+0000","last_undegraded":"2023-11-28T14:32:22.281524+0000","last_fullsized":"2023-11-28T14:32:22.281524+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941192,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":2187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.245219+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:32:22.245219+0000","last_peered":"2023-11-28T14:32:22.245219+0000","last_clean":"2023-11-28T14:32:22.245219+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:32:22.245219+0000","last_undegraded":"2023-11-28T14:32:22.245219+0000","last_fullsized":"2023-11-28T14:32:22.245219+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"19'3","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T14:32:21.225477+0000","last_change":"2023-11-28T14:32:21.235288+0000","last_active":"2023-11-28T14:32:21.225477+0000","last_peered":"2023-11-28T14:32:21.225477+0000","last_clean":"2023-11-28T14:32:21.225477+0000","last_became_active":"2023-11-28T14:32:09.311653+0000","last_became_peered":"2023-11-28T14:32:09.311653+0000","last_unstale":"2023-11-28T14:32:21.225477+0000","last_undegraded":"2023-11-28T14:32:21.225477+0000","last_fullsized":"2023-11-28T14:32:21.225477+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016807,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"23'16","reported_seq":40,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.510117+0000","last_change":"2023-11-28T14:32:17.473937+0000","last_active":"2023-11-28T14:32:20.510117+0000","last_peered":"2023-11-28T14:32:20.510117+0000","last_clean":"2023-11-28T14:32:20.510117+0000","last_became_active":"2023-11-28T14:32:09.316944+0000","last_became_peered":"2023-11-28T14:32:09.316944+0000","last_unstale":"2023-11-28T14:32:20.510117+0000","last_undegraded":"2023-11-28T14:32:20.510117+0000","last_fullsized":"2023-11-28T14:32:20.510117+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3193297,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":16,"num_write_kb":4613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T14:32:21.228743+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:32:21.228743+0000","last_peered":"2023-11-28T14:32:21.228743+0000","last_clean":"2023-11-28T14:32:21.228743+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:32:21.228743+0000","last_undegraded":"2023-11-28T14:32:21.228743+0000","last_fullsized":"2023-11-28T14:32:21.228743+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"23'13","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T14:32:21.227937+0000","last_change":"2023-11-28T14:32:21.234090+0000","last_active":"2023-11-28T14:32:21.227937+0000","last_peered":"2023-11-28T14:32:21.227937+0000","last_clean":"2023-11-28T14:32:21.227937+0000","last_became_active":"2023-11-28T14:32:09.309510+0000","last_became_peered":"2023-11-28T14:32:09.309510+0000","last_unstale":"2023-11-28T14:32:21.227937+0000","last_undegraded":"2023-11-28T14:32:21.227937+0000","last_fullsized":"2023-11-28T14:32:21.227937+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2386561,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3260,"num_write":12,"num_write_kb":3313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.284201+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:32:22.284201+0000","last_peered":"2023-11-28T14:32:22.284201+0000","last_clean":"2023-11-28T14:32:22.284201+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:32:22.284201+0000","last_undegraded":"2023-11-28T14:32:22.284201+0000","last_fullsized":"2023-11-28T14:32:22.284201+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"23'27","reported_seq":58,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-28T14:32:22.304334+0000","last_change":"2023-11-28T14:32:22.325131+0000","last_active":"2023-11-28T14:32:22.304334+0000","last_peered":"2023-11-28T14:32:22.304334+0000","last_clean":"2023-11-28T14:32:22.304334+0000","last_became_active":"2023-11-28T14:32:09.356991+0000","last_became_peered":"2023-11-28T14:32:09.356991+0000","last_unstale":"2023-11-28T14:32:22.304334+0000","last_undegraded":"2023-11-28T14:32:22.304334+0000","last_fullsized":"2023-11-28T14:32:22.304334+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151296,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":3102,"num_write":26,"num_write_kb":8152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T14:32:21.225295+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:32:21.225295+0000","last_peered":"2023-11-28T14:32:21.225295+0000","last_clean":"2023-11-28T14:32:21.225295+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:32:21.225295+0000","last_undegraded":"2023-11-28T14:32:21.225295+0000","last_fullsized":"2023-11-28T14:32:21.225295+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"19'6","reported_seq":42,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T14:32:21.229594+0000","last_change":"2023-11-28T14:32:21.235134+0000","last_active":"2023-11-28T14:32:21.229594+0000","last_peered":"2023-11-28T14:32:21.229594+0000","last_clean":"2023-11-28T14:32:21.229594+0000","last_became_active":"2023-11-28T14:32:09.337548+0000","last_became_peered":"2023-11-28T14:32:09.337548+0000","last_unstale":"2023-11-28T14:32:21.229594+0000","last_undegraded":"2023-11-28T14:32:21.229594+0000","last_fullsized":"2023-11-28T14:32:21.229594+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":9887,"num_write":8,"num_write_kb":2059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T14:32:20.503533+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:32:20.503533+0000","last_peered":"2023-11-28T14:32:20.503533+0000","last_clean":"2023-11-28T14:32:20.503533+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:32:20.503533+0000","last_undegraded":"2023-11-28T14:32:20.503533+0000","last_fullsized":"2023-11-28T14:32:20.503533+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":114067723,"num_objects":60,"num_object_clones":17,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":241,"num_read_kb":114206,"num_write":330,"num_write_kb":92793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":310,"ondisk_log_size":310,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75944,"kb_used_data":49120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94295896,"statfs":{"total":96636764160,"available":96558997504,"internally_reserved":0,"allocated":50298880,"data_stored":50793812,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705690,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81412,"kb_used_data":54588,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94290428,"statfs":{"total":96636764160,"available":96553398272,"internally_reserved":0,"allocated":55898112,"data_stored":56111818,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":58248,"kb_used_data":31424,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94313592,"statfs":{"total":96636764160,"available":96577118208,"internally_reserved":0,"allocated":32178176,"data_stored":32599026,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705690,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75372,"kb_used_data":47852,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94296468,"statfs":{"total":96636764160,"available":96559583232,"internally_reserved":0,"allocated":49000448,"data_stored":48537580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:32:27.896 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-11-28T14:32:27.896 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:32:27.896 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-11-28T14:32:27.897 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-11-28T14:32:28.009 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:28.004+0000 7f4147a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:28.012 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:28.011+0000 7f4147a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:28.025 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:28.025 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:28.025 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:28.025 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:28.025 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:28.025 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:28.025 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:28.026 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:28.026 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:28.026 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:28.026 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:28.026 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:28.027 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:28.028 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:28.028 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-11-28T14:32:28.142 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:28.137+0000 7f1d0391e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:28.145 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:28.144+0000 7f1d0391e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:28.157 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:28.157 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:28.157 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:28.158 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:28.158 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:28.158 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:28.158 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:28.158 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:28.158 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:28.158 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:28.158 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:28.158 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:28.160 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:28.161 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:28.161 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-11-28T14:32:28.253 INFO:tasks.rados.rados.0.smithi055.stdout:133: done (0 left) 2023-11-28T14:32:28.253 INFO:tasks.rados.rados.0.smithi055.stdout:134: read oid 14 snap 6 2023-11-28T14:32:28.254 INFO:tasks.rados.rados.0.smithi055.stdout:134: expect deleted 2023-11-28T14:32:28.254 INFO:tasks.rados.rados.0.smithi055.stdout:135: read oid 11 snap -1 2023-11-28T14:32:28.254 INFO:tasks.rados.rados.0.smithi055.stdout:135: expect (ObjNum 10 snap 0 seq_num 10) 2023-11-28T14:32:28.254 INFO:tasks.rados.rados.0.smithi055.stdout:136: snap_create 2023-11-28T14:32:28.277 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:28.271+0000 7ff3d222d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:28.281 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:28.280+0000 7ff3d222d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:28.296 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:28.296 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:28.297 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:28.297 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:28.297 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:28.297 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:28.297 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:28.297 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:28.297 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:28.297 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:28.297 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:28.297 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:28.300 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:28.300 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:28.301 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-11-28T14:32:28.315 INFO:tasks.rados.rados.0.smithi055.stdout:135: expect (ObjNum 10 snap 0 seq_num 10) 2023-11-28T14:32:28.415 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:28.409+0000 7f4148f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:28.418 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:28.417+0000 7f4148f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:28.431 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:28.431 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:28.431 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:28.431 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:28.431 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:28.431 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:28.431 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:28.431 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:28.431 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:28.431 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:28.432 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:28.432 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:28.434 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:29.256 INFO:tasks.rados.rados.0.smithi055.stdout:134: done (2 left) 2023-11-28T14:32:29.256 INFO:tasks.rados.rados.0.smithi055.stdout:135: done (1 left) 2023-11-28T14:32:29.256 INFO:tasks.rados.rados.0.smithi055.stdout:136: done (0 left) 2023-11-28T14:32:29.257 INFO:tasks.rados.rados.0.smithi055.stdout:137: write (excl) oid 3 current snap is 9 2023-11-28T14:32:29.257 INFO:tasks.rados.rados.0.smithi055.stdout:137: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2023-11-28T14:32:29.263 INFO:tasks.rados.rados.0.smithi055.stdout:137: writing smithi05583006-3 from 679016 to 1380811 tid 1 2023-11-28T14:32:29.269 INFO:tasks.rados.rados.0.smithi055.stdout:137: writing smithi05583006-3 from 1866533 to 2362557 tid 2 2023-11-28T14:32:29.274 INFO:tasks.rados.rados.0.smithi055.stdout:137: writing smithi05583006-3 from 2900635 to 3303278 tid 3 2023-11-28T14:32:29.274 INFO:tasks.rados.rados.0.smithi055.stdout:138: write (excl) oid 41 current snap is 9 2023-11-28T14:32:29.274 INFO:tasks.rados.rados.0.smithi055.stdout:138: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2023-11-28T14:32:29.282 INFO:tasks.rados.rados.0.smithi055.stdout:138: writing smithi05583006-41 from 695823 to 1472867 tid 1 2023-11-28T14:32:29.288 INFO:tasks.rados.rados.0.smithi055.stdout:138: writing smithi05583006-41 from 1925015 to 2564697 tid 2 2023-11-28T14:32:29.293 INFO:tasks.rados.rados.0.smithi055.stdout:138: writing smithi05583006-41 from 3328058 to 3495823 tid 3 2023-11-28T14:32:29.293 INFO:tasks.rados.rados.0.smithi055.stdout:139: write oid 46 current snap is 9 2023-11-28T14:32:29.294 INFO:tasks.rados.rados.0.smithi055.stdout:139: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2023-11-28T14:32:29.300 INFO:tasks.rados.rados.0.smithi055.stdout:139: writing smithi05583006-46 from 712630 to 1164923 tid 1 2023-11-28T14:32:29.306 INFO:tasks.rados.rados.0.smithi055.stdout:139: writing smithi05583006-46 from 1583497 to 2083190 tid 2 2023-11-28T14:32:29.312 INFO:tasks.rados.rados.0.smithi055.stdout:139: writing smithi05583006-46 from 2555481 to 2980668 tid 3 2023-11-28T14:32:29.312 INFO:tasks.rados.rados.0.smithi055.stdout:140: write oid 45 current snap is 9 2023-11-28T14:32:29.312 INFO:tasks.rados.rados.0.smithi055.stdout:140: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2023-11-28T14:32:29.316 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:32:29.319 INFO:tasks.rados.rados.0.smithi055.stdout:140: writing smithi05583006-45 from 729437 to 1256979 tid 1 2023-11-28T14:32:29.326 INFO:tasks.rados.rados.0.smithi055.stdout:140: writing smithi05583006-45 from 1925626 to 2568977 tid 2 2023-11-28T14:32:29.332 INFO:tasks.rados.rados.0.smithi055.stdout:140: writing smithi05583006-45 from 3266551 to 3529437 tid 3 2023-11-28T14:32:29.332 INFO:tasks.rados.rados.0.smithi055.stdout:141: write oid 17 current snap is 9 2023-11-28T14:32:29.332 INFO:tasks.rados.rados.0.smithi055.stdout:137: finishing write tid 1 to smithi05583006-3 2023-11-28T14:32:29.332 INFO:tasks.rados.rados.0.smithi055.stdout:141: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2023-11-28T14:32:29.340 INFO:tasks.rados.rados.0.smithi055.stdout:141: writing smithi05583006-17 from 746244 to 1465388 tid 1 2023-11-28T14:32:29.346 INFO:tasks.rados.rados.0.smithi055.stdout:141: writing smithi05583006-17 from 2100461 to 2603823 tid 2 2023-11-28T14:32:29.351 INFO:tasks.rados.rados.0.smithi055.stdout:141: writing smithi05583006-17 from 3126680 to 3546244 tid 3 2023-11-28T14:32:29.351 INFO:tasks.rados.rados.0.smithi055.stdout:142: read oid 49 snap -1 2023-11-28T14:32:29.352 INFO:tasks.rados.rados.0.smithi055.stdout:142: expect (ObjNum 79 snap 2 seq_num 79) 2023-11-28T14:32:29.352 INFO:tasks.rados.rados.0.smithi055.stdout:143: snap_remove snap 6 2023-11-28T14:32:29.445 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:29.442+0000 7fca52b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:29.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:29.447+0000 7fca52b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:30.274 INFO:tasks.rados.rados.0.smithi055.stdout:143: done (6 left) 2023-11-28T14:32:30.275 INFO:tasks.rados.rados.0.smithi055.stdout:144: read oid 24 snap -1 2023-11-28T14:32:30.275 INFO:tasks.rados.rados.0.smithi055.stdout:144: expect (ObjNum 23 snap 0 seq_num 23) 2023-11-28T14:32:30.275 INFO:tasks.rados.rados.0.smithi055.stdout:145: write (excl) oid 29 current snap is 9 2023-11-28T14:32:30.275 INFO:tasks.rados.rados.0.smithi055.stdout:145: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2023-11-28T14:32:30.283 INFO:tasks.rados.rados.0.smithi055.stdout:145: writing smithi05583006-29 from 763051 to 1557444 tid 1 2023-11-28T14:32:30.289 INFO:tasks.rados.rados.0.smithi055.stdout:145: writing smithi05583006-29 from 2158943 to 2805963 tid 2 2023-11-28T14:32:30.294 INFO:tasks.rados.rados.0.smithi055.stdout:145: writing smithi05583006-29 from 3437750 to 3563051 tid 3 2023-11-28T14:32:30.294 INFO:tasks.rados.rados.0.smithi055.stdout:146: delete oid 26 current snap is 9 2023-11-28T14:32:30.294 INFO:tasks.rados.rados.0.smithi055.stdout:137: finishing write tid 2 to smithi05583006-3 2023-11-28T14:32:30.294 INFO:tasks.rados.rados.0.smithi055.stdout:139: finishing write tid 1 to smithi05583006-46 2023-11-28T14:32:30.295 INFO:tasks.rados.rados.0.smithi055.stdout:140: finishing write tid 1 to smithi05583006-45 2023-11-28T14:32:30.295 INFO:tasks.rados.rados.0.smithi055.stdout:137: finishing write tid 3 to smithi05583006-3 2023-11-28T14:32:30.295 INFO:tasks.rados.rados.0.smithi055.stdout:141: finishing write tid 1 to smithi05583006-17 2023-11-28T14:32:30.295 INFO:tasks.rados.rados.0.smithi055.stdout:142: expect (ObjNum 79 snap 2 seq_num 79) 2023-11-28T14:32:30.502 INFO:tasks.rados.rados.0.smithi055.stdout:139: finishing write tid 2 to smithi05583006-46 2023-11-28T14:32:30.502 INFO:tasks.rados.rados.0.smithi055.stdout:140: finishing write tid 2 to smithi05583006-45 2023-11-28T14:32:30.502 INFO:tasks.rados.rados.0.smithi055.stdout:137: finishing write tid 4 to smithi05583006-3 2023-11-28T14:32:30.502 INFO:tasks.rados.rados.0.smithi055.stdout:137: finishing write tid 5 to smithi05583006-3 2023-11-28T14:32:30.502 INFO:tasks.rados.rados.0.smithi055.stdout:137: oid 3 updating version 0 to 63 2023-11-28T14:32:30.502 INFO:tasks.rados.rados.0.smithi055.stdout:137: oid 3 updating version 63 to 66 2023-11-28T14:32:30.502 INFO:tasks.rados.rados.0.smithi055.stdout:137: oid 3 version 66 is already newer than 65 2023-11-28T14:32:30.502 INFO:tasks.rados.rados.0.smithi055.stdout:137: oid 3 version 66 is already newer than 64 2023-11-28T14:32:30.502 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 3 v 66 (ObjNum 88 snap 9 seq_num 88) dirty exists 2023-11-28T14:32:30.503 INFO:tasks.rados.rados.0.smithi055.stdout:137: left oid 3 (ObjNum 88 snap 9 seq_num 88) 2023-11-28T14:32:30.503 INFO:tasks.rados.rados.0.smithi055.stdout:141: finishing write tid 2 to smithi05583006-17 2023-11-28T14:32:30.503 INFO:tasks.rados.rados.0.smithi055.stdout:137: done (8 left) 2023-11-28T14:32:30.503 INFO:tasks.rados.rados.0.smithi055.stdout:142: done (7 left) 2023-11-28T14:32:30.503 INFO:tasks.rados.rados.0.smithi055.stdout:146: done (6 left) 2023-11-28T14:32:30.503 INFO:tasks.rados.rados.0.smithi055.stdout:147: write (excl) oid 38 current snap is 9 2023-11-28T14:32:30.503 INFO:tasks.rados.rados.0.smithi055.stdout:139: finishing write tid 3 to smithi05583006-46 2023-11-28T14:32:30.503 INFO:tasks.rados.rados.0.smithi055.stdout:147: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2023-11-28T14:32:30.508 INFO:tasks.rados.rados.0.smithi055.stdout:147: writing smithi05583006-38 from 796665 to 1341556 tid 1 2023-11-28T14:32:30.514 INFO:tasks.rados.rados.0.smithi055.stdout:147: writing smithi05583006-38 from 1759554 to 2410243 tid 2 2023-11-28T14:32:30.522 INFO:tasks.rados.rados.0.smithi055.stdout:147: writing smithi05583006-38 from 2976243 to 3596665 tid 3 2023-11-28T14:32:30.522 INFO:tasks.rados.rados.0.smithi055.stdout:148: read oid 10 snap -1 2023-11-28T14:32:30.522 INFO:tasks.rados.rados.0.smithi055.stdout:148: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-28T14:32:30.522 INFO:tasks.rados.rados.0.smithi055.stdout:149: read oid 22 snap -1 2023-11-28T14:32:30.522 INFO:tasks.rados.rados.0.smithi055.stdout:149: expect (ObjNum 70 snap 1 seq_num 70) 2023-11-28T14:32:30.522 INFO:tasks.rados.rados.0.smithi055.stdout:150: snap_create 2023-11-28T14:32:30.522 INFO:tasks.rados.rados.0.smithi055.stdout:140: finishing write tid 3 to smithi05583006-45 2023-11-28T14:32:30.523 INFO:tasks.rados.rados.0.smithi055.stdout:138: finishing write tid 1 to smithi05583006-41 2023-11-28T14:32:30.523 INFO:tasks.rados.rados.0.smithi055.stdout:139: finishing write tid 4 to smithi05583006-46 2023-11-28T14:32:30.523 INFO:tasks.rados.rados.0.smithi055.stdout:139: finishing write tid 5 to smithi05583006-46 2023-11-28T14:32:30.523 INFO:tasks.rados.rados.0.smithi055.stdout:139: oid 46 updating version 0 to 62 2023-11-28T14:32:30.523 INFO:tasks.rados.rados.0.smithi055.stdout:139: oid 46 updating version 62 to 64 2023-11-28T14:32:30.523 INFO:tasks.rados.rados.0.smithi055.stdout:139: oid 46 version 64 is already newer than 63 2023-11-28T14:32:30.523 INFO:tasks.rados.rados.0.smithi055.stdout:139: oid 46 updating version 64 to 65 2023-11-28T14:32:30.523 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 46 v 65 (ObjNum 90 snap 9 seq_num 90) dirty exists 2023-11-28T14:32:30.523 INFO:tasks.rados.rados.0.smithi055.stdout:139: left oid 46 (ObjNum 90 snap 9 seq_num 90) 2023-11-28T14:32:30.523 INFO:tasks.rados.rados.0.smithi055.stdout:141: finishing write tid 3 to smithi05583006-17 2023-11-28T14:32:30.524 INFO:tasks.rados.rados.0.smithi055.stdout:140: finishing write tid 4 to smithi05583006-45 2023-11-28T14:32:30.524 INFO:tasks.rados.rados.0.smithi055.stdout:140: finishing write tid 5 to smithi05583006-45 2023-11-28T14:32:30.524 INFO:tasks.rados.rados.0.smithi055.stdout:140: oid 45 updating version 0 to 16 2023-11-28T14:32:30.524 INFO:tasks.rados.rados.0.smithi055.stdout:140: oid 45 version 16 is already newer than 15 2023-11-28T14:32:30.524 INFO:tasks.rados.rados.0.smithi055.stdout:140: oid 45 updating version 16 to 18 2023-11-28T14:32:30.524 INFO:tasks.rados.rados.0.smithi055.stdout:140: oid 45 version 18 is already newer than 17 2023-11-28T14:32:30.524 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 45 v 18 (ObjNum 91 snap 9 seq_num 91) dirty exists 2023-11-28T14:32:30.524 INFO:tasks.rados.rados.0.smithi055.stdout:140: left oid 45 (ObjNum 91 snap 9 seq_num 91) 2023-11-28T14:32:30.524 INFO:tasks.rados.rados.0.smithi055.stdout:138: finishing write tid 2 to smithi05583006-41 2023-11-28T14:32:30.524 INFO:tasks.rados.rados.0.smithi055.stdout:138: finishing write tid 3 to smithi05583006-41 2023-11-28T14:32:30.524 INFO:tasks.rados.rados.0.smithi055.stdout:138: finishing write tid 4 to smithi05583006-41 2023-11-28T14:32:30.525 INFO:tasks.rados.rados.0.smithi055.stdout:138: finishing write tid 5 to smithi05583006-41 2023-11-28T14:32:30.525 INFO:tasks.rados.rados.0.smithi055.stdout:138: oid 41 updating version 0 to 68 2023-11-28T14:32:30.525 INFO:tasks.rados.rados.0.smithi055.stdout:138: oid 41 updating version 68 to 71 2023-11-28T14:32:30.525 INFO:tasks.rados.rados.0.smithi055.stdout:138: oid 41 version 71 is already newer than 71 2023-11-28T14:32:30.525 INFO:tasks.rados.rados.0.smithi055.stdout:138: oid 41 version 71 is already newer than 71 2023-11-28T14:32:30.525 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 41 v 71 (ObjNum 89 snap 9 seq_num 89) dirty exists 2023-11-28T14:32:30.525 INFO:tasks.rados.rados.0.smithi055.stdout:138: left oid 41 (ObjNum 89 snap 9 seq_num 89) 2023-11-28T14:32:30.525 INFO:tasks.rados.rados.0.smithi055.stdout:141: finishing write tid 4 to smithi05583006-17 2023-11-28T14:32:30.525 INFO:tasks.rados.rados.0.smithi055.stdout:141: finishing write tid 5 to smithi05583006-17 2023-11-28T14:32:30.525 INFO:tasks.rados.rados.0.smithi055.stdout:141: oid 17 updating version 0 to 12 2023-11-28T14:32:30.526 INFO:tasks.rados.rados.0.smithi055.stdout:141: oid 17 updating version 12 to 15 2023-11-28T14:32:30.526 INFO:tasks.rados.rados.0.smithi055.stdout:141: oid 17 version 15 is already newer than 14 2023-11-28T14:32:30.526 INFO:tasks.rados.rados.0.smithi055.stdout:141: oid 17 version 15 is already newer than 13 2023-11-28T14:32:30.526 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 17 v 15 (ObjNum 92 snap 9 seq_num 92) dirty exists 2023-11-28T14:32:30.526 INFO:tasks.rados.rados.0.smithi055.stdout:141: left oid 17 (ObjNum 92 snap 9 seq_num 92) 2023-11-28T14:32:30.526 INFO:tasks.rados.rados.0.smithi055.stdout:144: expect (ObjNum 23 snap 0 seq_num 23) 2023-11-28T14:32:30.618 INFO:tasks.rados.rados.0.smithi055.stdout:145: finishing write tid 1 to smithi05583006-29 2023-11-28T14:32:30.618 INFO:tasks.rados.rados.0.smithi055.stdout:145: finishing write tid 2 to smithi05583006-29 2023-11-28T14:32:30.618 INFO:tasks.rados.rados.0.smithi055.stdout:145: finishing write tid 3 to smithi05583006-29 2023-11-28T14:32:30.618 INFO:tasks.rados.rados.0.smithi055.stdout:145: finishing write tid 4 to smithi05583006-29 2023-11-28T14:32:30.618 INFO:tasks.rados.rados.0.smithi055.stdout:145: finishing write tid 5 to smithi05583006-29 2023-11-28T14:32:30.618 INFO:tasks.rados.rados.0.smithi055.stdout:145: oid 29 updating version 0 to 14 2023-11-28T14:32:30.618 INFO:tasks.rados.rados.0.smithi055.stdout:145: oid 29 version 14 is already newer than 12 2023-11-28T14:32:30.618 INFO:tasks.rados.rados.0.smithi055.stdout:145: oid 29 version 14 is already newer than 13 2023-11-28T14:32:30.619 INFO:tasks.rados.rados.0.smithi055.stdout:145: oid 29 version 14 is already newer than 11 2023-11-28T14:32:30.619 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 29 v 14 (ObjNum 93 snap 9 seq_num 93) dirty exists 2023-11-28T14:32:30.619 INFO:tasks.rados.rados.0.smithi055.stdout:145: left oid 29 (ObjNum 93 snap 9 seq_num 93) 2023-11-28T14:32:30.619 INFO:tasks.rados.rados.0.smithi055.stdout:149: expect (ObjNum 70 snap 1 seq_num 70) 2023-11-28T14:32:30.829 INFO:tasks.rados.rados.0.smithi055.stdout:148: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-28T14:32:30.913 INFO:tasks.rados.rados.0.smithi055.stdout:147: finishing write tid 1 to smithi05583006-38 2023-11-28T14:32:30.914 INFO:tasks.rados.rados.0.smithi055.stdout:147: finishing write tid 2 to smithi05583006-38 2023-11-28T14:32:30.914 INFO:tasks.rados.rados.0.smithi055.stdout:147: finishing write tid 3 to smithi05583006-38 2023-11-28T14:32:30.914 INFO:tasks.rados.rados.0.smithi055.stdout:147: finishing write tid 4 to smithi05583006-38 2023-11-28T14:32:30.914 INFO:tasks.rados.rados.0.smithi055.stdout:147: finishing write tid 5 to smithi05583006-38 2023-11-28T14:32:30.914 INFO:tasks.rados.rados.0.smithi055.stdout:147: oid 38 updating version 0 to 16 2023-11-28T14:32:30.914 INFO:tasks.rados.rados.0.smithi055.stdout:147: oid 38 version 16 is already newer than 15 2023-11-28T14:32:30.914 INFO:tasks.rados.rados.0.smithi055.stdout:147: oid 38 updating version 16 to 18 2023-11-28T14:32:30.914 INFO:tasks.rados.rados.0.smithi055.stdout:147: oid 38 version 18 is already newer than 17 2023-11-28T14:32:30.914 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 38 v 18 (ObjNum 95 snap 9 seq_num 95) dirty exists 2023-11-28T14:32:30.914 INFO:tasks.rados.rados.0.smithi055.stdout:147: left oid 38 (ObjNum 95 snap 9 seq_num 95) 2023-11-28T14:32:31.282 INFO:tasks.rados.rados.0.smithi055.stdout:138: done (9 left) 2023-11-28T14:32:31.282 INFO:tasks.rados.rados.0.smithi055.stdout:139: done (8 left) 2023-11-28T14:32:31.282 INFO:tasks.rados.rados.0.smithi055.stdout:140: done (7 left) 2023-11-28T14:32:31.282 INFO:tasks.rados.rados.0.smithi055.stdout:141: done (6 left) 2023-11-28T14:32:31.282 INFO:tasks.rados.rados.0.smithi055.stdout:144: done (5 left) 2023-11-28T14:32:31.282 INFO:tasks.rados.rados.0.smithi055.stdout:145: done (4 left) 2023-11-28T14:32:31.282 INFO:tasks.rados.rados.0.smithi055.stdout:147: done (3 left) 2023-11-28T14:32:31.282 INFO:tasks.rados.rados.0.smithi055.stdout:148: done (2 left) 2023-11-28T14:32:31.283 INFO:tasks.rados.rados.0.smithi055.stdout:149: done (1 left) 2023-11-28T14:32:31.283 INFO:tasks.rados.rados.0.smithi055.stdout:150: done (0 left) 2023-11-28T14:32:31.283 INFO:tasks.rados.rados.0.smithi055.stdout:151: delete oid 48 current snap is 10 2023-11-28T14:32:31.317 INFO:tasks.rados.rados.0.smithi055.stdout:151: done (0 left) 2023-11-28T14:32:31.317 INFO:tasks.rados.rados.0.smithi055.stdout:152: read oid 45 snap 5 2023-11-28T14:32:31.317 INFO:tasks.rados.rados.0.smithi055.stdout:152: expect (ObjNum 44 snap 0 seq_num 44) 2023-11-28T14:32:31.317 INFO:tasks.rados.rados.0.smithi055.stdout:153: read oid 31 snap -1 2023-11-28T14:32:31.318 INFO:tasks.rados.rados.0.smithi055.stdout:153: expect (ObjNum 73 snap 2 seq_num 73) 2023-11-28T14:32:31.318 INFO:tasks.rados.rados.0.smithi055.stdout:154: snap_remove snap 8 2023-11-28T14:32:31.587 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:32:31.699 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:31.696+0000 7f51c5d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:31.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:31.700+0000 7f51c5d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:32.282 INFO:tasks.rados.rados.0.smithi055.stdout:154: done (2 left) 2023-11-28T14:32:32.283 INFO:tasks.rados.rados.0.smithi055.stdout:155: write (excl) oid 4 current snap is 10 2023-11-28T14:32:32.283 INFO:tasks.rados.rados.0.smithi055.stdout:155: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2023-11-28T14:32:32.288 INFO:tasks.rados.rados.0.smithi055.stdout:155: writing smithi05583006-4 from 430279 to 1125668 tid 1 2023-11-28T14:32:32.296 INFO:tasks.rados.rados.0.smithi055.stdout:155: writing smithi05583006-4 from 1876518 to 2530876 tid 2 2023-11-28T14:32:32.302 INFO:tasks.rados.rados.0.smithi055.stdout:155: writing smithi05583006-4 from 3031089 to 3567886 tid 3 2023-11-28T14:32:32.302 INFO:tasks.rados.rados.0.smithi055.stdout:156: snap_remove snap 9 2023-11-28T14:32:33.301 INFO:tasks.rados.rados.0.smithi055.stdout:156: done (3 left) 2023-11-28T14:32:33.301 INFO:tasks.rados.rados.0.smithi055.stdout:157: snap_create 2023-11-28T14:32:33.301 INFO:tasks.rados.rados.0.smithi055.stdout:152: expect (ObjNum 44 snap 0 seq_num 44) 2023-11-28T14:32:33.406 INFO:tasks.rados.rados.0.smithi055.stdout:153: expect (ObjNum 73 snap 2 seq_num 73) 2023-11-28T14:32:33.435 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:32:33.435 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:32:33.435 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:32:33.548 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:33.542+0000 7fbbe9111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:33.550 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:33.549+0000 7fbbe9111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:33.618 INFO:tasks.rados.rados.0.smithi055.stdout:155: finishing write tid 1 to smithi05583006-4 2023-11-28T14:32:33.618 INFO:tasks.rados.rados.0.smithi055.stdout:155: finishing write tid 2 to smithi05583006-4 2023-11-28T14:32:33.618 INFO:tasks.rados.rados.0.smithi055.stdout:155: finishing write tid 3 to smithi05583006-4 2023-11-28T14:32:33.618 INFO:tasks.rados.rados.0.smithi055.stdout:155: finishing write tid 4 to smithi05583006-4 2023-11-28T14:32:33.618 INFO:tasks.rados.rados.0.smithi055.stdout:155: finishing write tid 5 to smithi05583006-4 2023-11-28T14:32:33.618 INFO:tasks.rados.rados.0.smithi055.stdout:155: oid 4 updating version 0 to 18 2023-11-28T14:32:33.618 INFO:tasks.rados.rados.0.smithi055.stdout:155: oid 4 updating version 18 to 20 2023-11-28T14:32:33.618 INFO:tasks.rados.rados.0.smithi055.stdout:155: oid 4 updating version 20 to 21 2023-11-28T14:32:33.618 INFO:tasks.rados.rados.0.smithi055.stdout:155: oid 4 version 21 is already newer than 19 2023-11-28T14:32:33.619 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 4 v 21 (ObjNum 97 snap 10 seq_num 97) dirty exists 2023-11-28T14:32:33.619 INFO:tasks.rados.rados.0.smithi055.stdout:155: left oid 4 (ObjNum 97 snap 10 seq_num 97) 2023-11-28T14:32:33.735 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:33.736 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:32:33.747 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":49,"stamp":"2023-11-28T14:32:33.298954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114036265,"num_objects":67,"num_object_clones":22,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":360,"num_read_kb":131223,"num_write":487,"num_write_kb":101141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":437,"ondisk_log_size":437,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":295544,"kb_used_data":188248,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377191816,"statfs":{"total":386547056640,"available":386244419584,"internally_reserved":0,"allocated":192765952,"data_stored":195634546,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-672247,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":10,"num_read_kb":8239,"num_write":15,"num_write_kb":4644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.044915"},"pg_stats":[{"pgid":"3.f","version":"35'75","reported_seq":131,"reported_epoch":37,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:32:32.487086+0000","last_change":"2023-11-28T14:32:32.487086+0000","last_active":"2023-11-28T14:32:32.487086+0000","last_peered":"2023-11-28T14:32:32.487086+0000","last_clean":"2023-11-28T14:32:32.487086+0000","last_became_active":"2023-11-28T14:32:09.321992+0000","last_became_peered":"2023-11-28T14:32:09.321992+0000","last_unstale":"2023-11-28T14:32:32.487086+0000","last_undegraded":"2023-11-28T14:32:32.487086+0000","last_fullsized":"2023-11-28T14:32:32.487086+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16672379,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":47,"num_read_kb":24713,"num_write":71,"num_write_kb":20294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"19'3","reported_seq":46,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T14:32:28.273358+0000","last_change":"2023-11-28T14:32:28.283937+0000","last_active":"2023-11-28T14:32:28.273358+0000","last_peered":"2023-11-28T14:32:28.273358+0000","last_clean":"2023-11-28T14:32:28.273358+0000","last_became_active":"2023-11-28T14:32:09.308848+0000","last_became_peered":"2023-11-28T14:32:09.308848+0000","last_unstale":"2023-11-28T14:32:28.273358+0000","last_undegraded":"2023-11-28T14:32:28.273358+0000","last_fullsized":"2023-11-28T14:32:28.273358+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":6433,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'15","reported_seq":66,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T14:32:28.282929+0000","last_change":"2023-11-28T14:32:28.296033+0000","last_active":"2023-11-28T14:32:28.282929+0000","last_peered":"2023-11-28T14:32:28.282929+0000","last_clean":"2023-11-28T14:32:28.282929+0000","last_became_active":"2023-11-28T14:32:09.337560+0000","last_became_peered":"2023-11-28T14:32:09.337560+0000","last_unstale":"2023-11-28T14:32:28.282929+0000","last_undegraded":"2023-11-28T14:32:28.282929+0000","last_fullsized":"2023-11-28T14:32:28.282929+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7848770,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":24013,"num_write":16,"num_write_kb":4334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"29'26","reported_seq":78,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T14:32:32.479686+0000","last_change":"2023-11-28T14:32:32.491000+0000","last_active":"2023-11-28T14:32:32.479686+0000","last_peered":"2023-11-28T14:32:32.479686+0000","last_clean":"2023-11-28T14:32:32.479686+0000","last_became_active":"2023-11-28T14:32:09.344782+0000","last_became_peered":"2023-11-28T14:32:09.344782+0000","last_unstale":"2023-11-28T14:32:32.479686+0000","last_undegraded":"2023-11-28T14:32:32.479686+0000","last_fullsized":"2023-11-28T14:32:32.479686+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11058874,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":9912,"num_write":26,"num_write_kb":7877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'23","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-28T14:32:26.264403+0000","last_change":"2023-11-28T14:32:24.268020+0000","last_active":"2023-11-28T14:32:26.264403+0000","last_peered":"2023-11-28T14:32:26.264403+0000","last_clean":"2023-11-28T14:32:26.264403+0000","last_became_active":"2023-11-28T14:32:09.326269+0000","last_became_peered":"2023-11-28T14:32:09.326269+0000","last_unstale":"2023-11-28T14:32:26.264403+0000","last_undegraded":"2023-11-28T14:32:26.264403+0000","last_fullsized":"2023-11-28T14:32:26.264403+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8470629,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":7024,"num_write":26,"num_write_kb":7179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"35'14","reported_seq":67,"reported_epoch":37,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:32:32.477925+0000","last_change":"2023-11-28T14:32:32.477925+0000","last_active":"2023-11-28T14:32:32.477925+0000","last_peered":"2023-11-28T14:32:32.477925+0000","last_clean":"2023-11-28T14:32:32.477925+0000","last_became_active":"2023-11-28T14:32:09.308846+0000","last_became_peered":"2023-11-28T14:32:09.308846+0000","last_unstale":"2023-11-28T14:32:32.477925+0000","last_undegraded":"2023-11-28T14:32:32.477925+0000","last_fullsized":"2023-11-28T14:32:32.477925+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8084068,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":5567,"num_write":17,"num_write_kb":4623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"35'23","reported_seq":74,"reported_epoch":37,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:32:32.477129+0000","last_change":"2023-11-28T14:32:32.477129+0000","last_active":"2023-11-28T14:32:32.477129+0000","last_peered":"2023-11-28T14:32:32.477129+0000","last_clean":"2023-11-28T14:32:32.477129+0000","last_became_active":"2023-11-28T14:32:09.316660+0000","last_became_peered":"2023-11-28T14:32:09.316660+0000","last_unstale":"2023-11-28T14:32:32.477129+0000","last_undegraded":"2023-11-28T14:32:32.477129+0000","last_fullsized":"2023-11-28T14:32:32.477129+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226911,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":11,"num_read_kb":7663,"num_write":23,"num_write_kb":6709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'13","reported_seq":58,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T14:32:28.278884+0000","last_change":"2023-11-28T14:32:28.295754+0000","last_active":"2023-11-28T14:32:28.278884+0000","last_peered":"2023-11-28T14:32:28.278884+0000","last_clean":"2023-11-28T14:32:28.278884+0000","last_became_active":"2023-11-28T14:32:09.358025+0000","last_became_peered":"2023-11-28T14:32:09.358025+0000","last_unstale":"2023-11-28T14:32:28.278884+0000","last_undegraded":"2023-11-28T14:32:28.278884+0000","last_fullsized":"2023-11-28T14:32:28.278884+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7075648,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":3594,"num_write":16,"num_write_kb":3963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":172,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-28T14:32:26.263548+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:32:26.263548+0000","last_peered":"2023-11-28T14:32:26.263548+0000","last_clean":"2023-11-28T14:32:26.263548+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:32:26.263548+0000","last_undegraded":"2023-11-28T14:32:26.263548+0000","last_fullsized":"2023-11-28T14:32:26.263548+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T14:32:28.284352+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:32:28.284352+0000","last_peered":"2023-11-28T14:32:28.284352+0000","last_clean":"2023-11-28T14:32:28.284352+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:32:28.284352+0000","last_undegraded":"2023-11-28T14:32:28.284352+0000","last_fullsized":"2023-11-28T14:32:28.284352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"23'60","reported_seq":116,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T14:32:28.298868+0000","last_change":"2023-11-28T14:32:28.295307+0000","last_active":"2023-11-28T14:32:28.298868+0000","last_peered":"2023-11-28T14:32:28.298868+0000","last_clean":"2023-11-28T14:32:28.298868+0000","last_became_active":"2023-11-28T14:32:09.335805+0000","last_became_peered":"2023-11-28T14:32:09.335805+0000","last_unstale":"2023-11-28T14:32:28.298868+0000","last_undegraded":"2023-11-28T14:32:28.298868+0000","last_fullsized":"2023-11-28T14:32:28.298868+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19226977,"num_objects":11,"num_object_clones":4,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":25953,"num_write":60,"num_write_kb":16715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":43,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T14:32:32.482409+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:32:32.482409+0000","last_peered":"2023-11-28T14:32:32.482409+0000","last_clean":"2023-11-28T14:32:32.482409+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:32:32.482409+0000","last_undegraded":"2023-11-28T14:32:32.482409+0000","last_fullsized":"2023-11-28T14:32:32.482409+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"18'13","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-28T14:32:26.262315+0000","last_change":"2023-11-28T14:32:24.264405+0000","last_active":"2023-11-28T14:32:26.262315+0000","last_peered":"2023-11-28T14:32:26.262315+0000","last_clean":"2023-11-28T14:32:26.262315+0000","last_became_active":"2023-11-28T14:32:09.350412+0000","last_became_peered":"2023-11-28T14:32:09.350412+0000","last_unstale":"2023-11-28T14:32:26.262315+0000","last_undegraded":"2023-11-28T14:32:26.262315+0000","last_fullsized":"2023-11-28T14:32:26.262315+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7680700,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":16,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T14:32:28.290317+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:32:28.290317+0000","last_peered":"2023-11-28T14:32:28.290317+0000","last_clean":"2023-11-28T14:32:28.290317+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:32:28.290317+0000","last_undegraded":"2023-11-28T14:32:28.290317+0000","last_fullsized":"2023-11-28T14:32:28.290317+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'10","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-28T14:32:26.261751+0000","last_change":"2023-11-28T14:32:24.265688+0000","last_active":"2023-11-28T14:32:26.261751+0000","last_peered":"2023-11-28T14:32:26.261751+0000","last_clean":"2023-11-28T14:32:26.261751+0000","last_became_active":"2023-11-28T14:32:09.343664+0000","last_became_peered":"2023-11-28T14:32:09.343664+0000","last_unstale":"2023-11-28T14:32:26.261751+0000","last_undegraded":"2023-11-28T14:32:26.261751+0000","last_fullsized":"2023-11-28T14:32:26.261751+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941192,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":2187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":43,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T14:32:32.484054+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:32:32.484054+0000","last_peered":"2023-11-28T14:32:32.484054+0000","last_clean":"2023-11-28T14:32:32.484054+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:32:32.484054+0000","last_undegraded":"2023-11-28T14:32:32.484054+0000","last_fullsized":"2023-11-28T14:32:32.484054+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"19'3","reported_seq":43,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T14:32:28.277798+0000","last_change":"2023-11-28T14:32:28.285231+0000","last_active":"2023-11-28T14:32:28.277798+0000","last_peered":"2023-11-28T14:32:28.277798+0000","last_clean":"2023-11-28T14:32:28.277798+0000","last_became_active":"2023-11-28T14:32:09.311653+0000","last_became_peered":"2023-11-28T14:32:09.311653+0000","last_unstale":"2023-11-28T14:32:28.277798+0000","last_undegraded":"2023-11-28T14:32:28.277798+0000","last_fullsized":"2023-11-28T14:32:28.277798+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016807,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"23'16","reported_seq":54,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T14:32:28.273176+0000","last_change":"2023-11-28T14:32:28.295184+0000","last_active":"2023-11-28T14:32:28.273176+0000","last_peered":"2023-11-28T14:32:28.273176+0000","last_clean":"2023-11-28T14:32:28.273176+0000","last_became_active":"2023-11-28T14:32:09.316944+0000","last_became_peered":"2023-11-28T14:32:09.316944+0000","last_unstale":"2023-11-28T14:32:28.273176+0000","last_undegraded":"2023-11-28T14:32:28.273176+0000","last_fullsized":"2023-11-28T14:32:28.273176+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3193297,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":16,"num_write_kb":4613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T14:32:28.274683+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:32:28.274683+0000","last_peered":"2023-11-28T14:32:28.274683+0000","last_clean":"2023-11-28T14:32:28.274683+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:32:28.274683+0000","last_undegraded":"2023-11-28T14:32:28.274683+0000","last_fullsized":"2023-11-28T14:32:28.274683+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"23'13","reported_seq":54,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T14:32:28.276890+0000","last_change":"2023-11-28T14:32:28.279689+0000","last_active":"2023-11-28T14:32:28.276890+0000","last_peered":"2023-11-28T14:32:28.276890+0000","last_clean":"2023-11-28T14:32:28.276890+0000","last_became_active":"2023-11-28T14:32:09.309510+0000","last_became_peered":"2023-11-28T14:32:09.309510+0000","last_unstale":"2023-11-28T14:32:28.276890+0000","last_undegraded":"2023-11-28T14:32:28.276890+0000","last_fullsized":"2023-11-28T14:32:28.276890+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2386561,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3260,"num_write":12,"num_write_kb":3313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-28T14:32:26.261226+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:32:26.261226+0000","last_peered":"2023-11-28T14:32:26.261226+0000","last_clean":"2023-11-28T14:32:26.261226+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:32:26.261226+0000","last_undegraded":"2023-11-28T14:32:26.261226+0000","last_fullsized":"2023-11-28T14:32:26.261226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"23'27","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-28T14:32:26.256863+0000","last_change":"2023-11-28T14:32:24.267842+0000","last_active":"2023-11-28T14:32:26.256863+0000","last_peered":"2023-11-28T14:32:26.256863+0000","last_clean":"2023-11-28T14:32:26.256863+0000","last_became_active":"2023-11-28T14:32:09.356991+0000","last_became_peered":"2023-11-28T14:32:09.356991+0000","last_unstale":"2023-11-28T14:32:26.256863+0000","last_undegraded":"2023-11-28T14:32:26.256863+0000","last_fullsized":"2023-11-28T14:32:26.256863+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151296,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":3102,"num_write":26,"num_write_kb":8152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":39,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T14:32:28.280446+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:32:28.280446+0000","last_peered":"2023-11-28T14:32:28.280446+0000","last_clean":"2023-11-28T14:32:28.280446+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:32:28.280446+0000","last_undegraded":"2023-11-28T14:32:28.280446+0000","last_fullsized":"2023-11-28T14:32:28.280446+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"19'6","reported_seq":53,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T14:32:28.276886+0000","last_change":"2023-11-28T14:32:28.284522+0000","last_active":"2023-11-28T14:32:28.276886+0000","last_peered":"2023-11-28T14:32:28.276886+0000","last_clean":"2023-11-28T14:32:28.276886+0000","last_became_active":"2023-11-28T14:32:09.337548+0000","last_became_peered":"2023-11-28T14:32:09.337548+0000","last_unstale":"2023-11-28T14:32:28.276886+0000","last_undegraded":"2023-11-28T14:32:28.276886+0000","last_fullsized":"2023-11-28T14:32:28.276886+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":9887,"num_write":8,"num_write_kb":2059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":35,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T14:32:28.290889+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:32:28.290889+0000","last_peered":"2023-11-28T14:32:28.290889+0000","last_clean":"2023-11-28T14:32:28.290889+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:32:28.290889+0000","last_undegraded":"2023-11-28T14:32:28.290889+0000","last_fullsized":"2023-11-28T14:32:28.290889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":113445897,"num_objects":65,"num_object_clones":22,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":257,"num_read_kb":131132,"num_write":350,"num_write_kb":99247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":340,"ondisk_log_size":340,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84340,"kb_used_data":57516,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94287500,"statfs":{"total":96636764160,"available":96550400000,"internally_reserved":0,"allocated":58896384,"data_stored":61994149,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705695,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79824,"kb_used_data":53000,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94292016,"statfs":{"total":96636764160,"available":96555024384,"internally_reserved":0,"allocated":54272000,"data_stored":54290945,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56672,"kb_used_data":29848,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94315168,"statfs":{"total":96636764160,"available":96578732032,"internally_reserved":0,"allocated":30564352,"data_stored":30787368,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705695,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74708,"kb_used_data":47884,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94297132,"statfs":{"total":96636764160,"available":96560263168,"internally_reserved":0,"allocated":49033216,"data_stored":48562084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:32:33.773 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:32:33.885 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:33.882+0000 7f938c4de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:33.888 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:33.886+0000 7f938c4de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:34.309 INFO:tasks.rados.rados.0.smithi055.stdout:152: done (3 left) 2023-11-28T14:32:34.309 INFO:tasks.rados.rados.0.smithi055.stdout:153: done (2 left) 2023-11-28T14:32:34.309 INFO:tasks.rados.rados.0.smithi055.stdout:155: done (1 left) 2023-11-28T14:32:34.309 INFO:tasks.rados.rados.0.smithi055.stdout:157: done (0 left) 2023-11-28T14:32:34.310 INFO:tasks.rados.rados.0.smithi055.stdout:158: snap_remove snap 10 2023-11-28T14:32:35.321 INFO:tasks.rados.rados.0.smithi055.stdout:158: done (0 left) 2023-11-28T14:32:35.321 INFO:tasks.rados.rados.0.smithi055.stdout:159: delete oid 21 current snap is 11 2023-11-28T14:32:35.350 INFO:tasks.rados.rados.0.smithi055.stdout:159: done (0 left) 2023-11-28T14:32:35.350 INFO:tasks.rados.rados.0.smithi055.stdout:160: delete oid 48 current snap is 11 2023-11-28T14:32:35.383 INFO:tasks.rados.rados.0.smithi055.stdout:160: done (0 left) 2023-11-28T14:32:35.383 INFO:tasks.rados.rados.0.smithi055.stdout:161: delete oid 11 current snap is 11 2023-11-28T14:32:35.405 INFO:tasks.rados.rados.0.smithi055.stdout:161: done (0 left) 2023-11-28T14:32:35.406 INFO:tasks.rados.rados.0.smithi055.stdout:162: write (excl) oid 44 current snap is 11 2023-11-28T14:32:35.406 INFO:tasks.rados.rados.0.smithi055.stdout:162: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2023-11-28T14:32:35.410 INFO:tasks.rados.rados.0.smithi055.stdout:162: writing smithi05583006-44 from 497507 to 1210245 tid 1 2023-11-28T14:32:35.417 INFO:tasks.rados.rados.0.smithi055.stdout:162: writing smithi05583006-44 from 1710446 to 2372142 tid 2 2023-11-28T14:32:35.424 INFO:tasks.rados.rados.0.smithi055.stdout:162: writing smithi05583006-44 from 3140781 to 3697507 tid 3 2023-11-28T14:32:35.424 INFO:tasks.rados.rados.0.smithi055.stdout:163: write oid 2 current snap is 11 2023-11-28T14:32:35.425 INFO:tasks.rados.rados.0.smithi055.stdout:163: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2023-11-28T14:32:35.431 INFO:tasks.rados.rados.0.smithi055.stdout:163: writing smithi05583006-2 from 514314 to 1302301 tid 1 2023-11-28T14:32:35.437 INFO:tasks.rados.rados.0.smithi055.stdout:163: writing smithi05583006-2 from 1768928 to 2174282 tid 2 2023-11-28T14:32:35.444 INFO:tasks.rados.rados.0.smithi055.stdout:163: writing smithi05583006-2 from 2768204 to 3277714 tid 3 2023-11-28T14:32:35.444 INFO:tasks.rados.rados.0.smithi055.stdout:164: read oid 21 snap -1 2023-11-28T14:32:35.444 INFO:tasks.rados.rados.0.smithi055.stdout:164: expect deleted 2023-11-28T14:32:35.444 INFO:tasks.rados.rados.0.smithi055.stdout:165: snap_remove snap 5 2023-11-28T14:32:35.959 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:32:36.071 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:36.068+0000 7f2c98206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:36.074 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:36.072+0000 7f2c98206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:36.325 INFO:tasks.rados.rados.0.smithi055.stdout:165: done (3 left) 2023-11-28T14:32:36.326 INFO:tasks.rados.rados.0.smithi055.stdout:166: write (excl) oid 33 current snap is 11 2023-11-28T14:32:36.326 INFO:tasks.rados.rados.0.smithi055.stdout:166: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2023-11-28T14:32:36.330 INFO:tasks.rados.rados.0.smithi055.stdout:166: writing smithi05583006-33 from 531121 to 994357 tid 1 2023-11-28T14:32:36.338 INFO:tasks.rados.rados.0.smithi055.stdout:166: writing smithi05583006-33 from 1711057 to 2376422 tid 2 2023-11-28T14:32:36.346 INFO:tasks.rados.rados.0.smithi055.stdout:166: writing smithi05583006-33 from 3079274 to 3731121 tid 3 2023-11-28T14:32:36.346 INFO:tasks.rados.rados.0.smithi055.stdout:167: read oid 19 snap -1 2023-11-28T14:32:36.346 INFO:tasks.rados.rados.0.smithi055.stdout:167: expect (ObjNum 18 snap 0 seq_num 18) 2023-11-28T14:32:36.346 INFO:tasks.rados.rados.0.smithi055.stdout:168: snap_create 2023-11-28T14:32:36.347 INFO:tasks.rados.rados.0.smithi055.stdout:162: finishing write tid 1 to smithi05583006-44 2023-11-28T14:32:36.347 INFO:tasks.rados.rados.0.smithi055.stdout:162: finishing write tid 2 to smithi05583006-44 2023-11-28T14:32:36.347 INFO:tasks.rados.rados.0.smithi055.stdout:162: finishing write tid 3 to smithi05583006-44 2023-11-28T14:32:36.347 INFO:tasks.rados.rados.0.smithi055.stdout:162: finishing write tid 4 to smithi05583006-44 2023-11-28T14:32:36.347 INFO:tasks.rados.rados.0.smithi055.stdout:162: finishing write tid 5 to smithi05583006-44 2023-11-28T14:32:36.347 INFO:tasks.rados.rados.0.smithi055.stdout:162: oid 44 updating version 0 to 84 2023-11-28T14:32:36.347 INFO:tasks.rados.rados.0.smithi055.stdout:162: oid 44 version 84 is already newer than 82 2023-11-28T14:32:36.347 INFO:tasks.rados.rados.0.smithi055.stdout:162: oid 44 version 84 is already newer than 83 2023-11-28T14:32:36.347 INFO:tasks.rados.rados.0.smithi055.stdout:162: oid 44 version 84 is already newer than 81 2023-11-28T14:32:36.347 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 44 v 84 (ObjNum 101 snap 11 seq_num 101) dirty exists 2023-11-28T14:32:36.347 INFO:tasks.rados.rados.0.smithi055.stdout:162: left oid 44 (ObjNum 101 snap 11 seq_num 101) 2023-11-28T14:32:36.347 INFO:tasks.rados.rados.0.smithi055.stdout:163: finishing write tid 1 to smithi05583006-2 2023-11-28T14:32:36.347 INFO:tasks.rados.rados.0.smithi055.stdout:163: finishing write tid 2 to smithi05583006-2 2023-11-28T14:32:36.348 INFO:tasks.rados.rados.0.smithi055.stdout:163: finishing write tid 3 to smithi05583006-2 2023-11-28T14:32:36.348 INFO:tasks.rados.rados.0.smithi055.stdout:163: finishing write tid 4 to smithi05583006-2 2023-11-28T14:32:36.348 INFO:tasks.rados.rados.0.smithi055.stdout:163: finishing write tid 5 to smithi05583006-2 2023-11-28T14:32:36.348 INFO:tasks.rados.rados.0.smithi055.stdout:163: oid 2 updating version 0 to 89 2023-11-28T14:32:36.348 INFO:tasks.rados.rados.0.smithi055.stdout:163: oid 2 version 89 is already newer than 86 2023-11-28T14:32:36.348 INFO:tasks.rados.rados.0.smithi055.stdout:163: oid 2 version 89 is already newer than 88 2023-11-28T14:32:36.348 INFO:tasks.rados.rados.0.smithi055.stdout:163: oid 2 version 89 is already newer than 87 2023-11-28T14:32:36.348 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 2 v 89 (ObjNum 102 snap 11 seq_num 102) dirty exists 2023-11-28T14:32:36.348 INFO:tasks.rados.rados.0.smithi055.stdout:163: left oid 2 (ObjNum 102 snap 11 seq_num 102) 2023-11-28T14:32:36.439 INFO:tasks.rados.rados.0.smithi055.stdout:167: expect (ObjNum 18 snap 0 seq_num 18) 2023-11-28T14:32:36.534 INFO:tasks.rados.rados.0.smithi055.stdout:166: finishing write tid 1 to smithi05583006-33 2023-11-28T14:32:36.534 INFO:tasks.rados.rados.0.smithi055.stdout:166: finishing write tid 2 to smithi05583006-33 2023-11-28T14:32:36.534 INFO:tasks.rados.rados.0.smithi055.stdout:166: finishing write tid 3 to smithi05583006-33 2023-11-28T14:32:36.541 INFO:tasks.rados.rados.0.smithi055.stdout:166: finishing write tid 4 to smithi05583006-33 2023-11-28T14:32:36.547 INFO:tasks.rados.rados.0.smithi055.stdout:166: finishing write tid 5 to smithi05583006-33 2023-11-28T14:32:36.547 INFO:tasks.rados.rados.0.smithi055.stdout:166: oid 33 updating version 0 to 78 2023-11-28T14:32:36.547 INFO:tasks.rados.rados.0.smithi055.stdout:166: oid 33 version 78 is already newer than 78 2023-11-28T14:32:36.547 INFO:tasks.rados.rados.0.smithi055.stdout:166: oid 33 version 78 is already newer than 76 2023-11-28T14:32:36.547 INFO:tasks.rados.rados.0.smithi055.stdout:166: oid 33 updating version 78 to 79 2023-11-28T14:32:36.547 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 33 v 79 (ObjNum 103 snap 11 seq_num 103) dirty exists 2023-11-28T14:32:36.547 INFO:tasks.rados.rados.0.smithi055.stdout:166: left oid 33 (ObjNum 103 snap 11 seq_num 103) 2023-11-28T14:32:37.345 INFO:tasks.rados.rados.0.smithi055.stdout:162: done (5 left) 2023-11-28T14:32:37.345 INFO:tasks.rados.rados.0.smithi055.stdout:163: done (4 left) 2023-11-28T14:32:37.345 INFO:tasks.rados.rados.0.smithi055.stdout:164: done (3 left) 2023-11-28T14:32:37.345 INFO:tasks.rados.rados.0.smithi055.stdout:166: done (2 left) 2023-11-28T14:32:37.345 INFO:tasks.rados.rados.0.smithi055.stdout:167: done (1 left) 2023-11-28T14:32:37.345 INFO:tasks.rados.rados.0.smithi055.stdout:168: done (0 left) 2023-11-28T14:32:37.345 INFO:tasks.rados.rados.0.smithi055.stdout:169: read oid 1 snap -1 2023-11-28T14:32:37.345 INFO:tasks.rados.rados.0.smithi055.stdout:169: expect (ObjNum 0 snap 0 seq_num 0) 2023-11-28T14:32:37.345 INFO:tasks.rados.rados.0.smithi055.stdout:170: write oid 40 current snap is 12 2023-11-28T14:32:37.346 INFO:tasks.rados.rados.0.smithi055.stdout:170: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2023-11-28T14:32:37.350 INFO:tasks.rados.rados.0.smithi055.stdout:170: writing smithi05583006-40 from 547928 to 1086413 tid 1 2023-11-28T14:32:37.356 INFO:tasks.rados.rados.0.smithi055.stdout:170: writing smithi05583006-40 from 1769539 to 2178562 tid 2 2023-11-28T14:32:37.362 INFO:tasks.rados.rados.0.smithi055.stdout:170: writing smithi05583006-40 from 2706697 to 3238751 tid 3 2023-11-28T14:32:37.362 INFO:tasks.rados.rados.0.smithi055.stdout:171: snap_create 2023-11-28T14:32:37.405 INFO:tasks.rados.rados.0.smithi055.stdout:170: finishing write tid 1 to smithi05583006-40 2023-11-28T14:32:37.408 INFO:tasks.rados.rados.0.smithi055.stdout:169: expect (ObjNum 0 snap 0 seq_num 0) 2023-11-28T14:32:37.486 INFO:tasks.rados.rados.0.smithi055.stdout:170: finishing write tid 2 to smithi05583006-40 2023-11-28T14:32:37.486 INFO:tasks.rados.rados.0.smithi055.stdout:170: finishing write tid 3 to smithi05583006-40 2023-11-28T14:32:37.486 INFO:tasks.rados.rados.0.smithi055.stdout:170: finishing write tid 4 to smithi05583006-40 2023-11-28T14:32:37.486 INFO:tasks.rados.rados.0.smithi055.stdout:170: finishing write tid 5 to smithi05583006-40 2023-11-28T14:32:37.486 INFO:tasks.rados.rados.0.smithi055.stdout:170: oid 40 updating version 0 to 6 2023-11-28T14:32:37.486 INFO:tasks.rados.rados.0.smithi055.stdout:170: oid 40 updating version 6 to 8 2023-11-28T14:32:37.486 INFO:tasks.rados.rados.0.smithi055.stdout:170: oid 40 version 8 is already newer than 8 2023-11-28T14:32:37.486 INFO:tasks.rados.rados.0.smithi055.stdout:170: oid 40 version 8 is already newer than 5 2023-11-28T14:32:37.486 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 40 v 8 (ObjNum 104 snap 12 seq_num 104) dirty exists 2023-11-28T14:32:37.487 INFO:tasks.rados.rados.0.smithi055.stdout:170: left oid 40 (ObjNum 104 snap 12 seq_num 104) 2023-11-28T14:32:38.144 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:32:38.271 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:38.268+0000 7f124a806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:38.274 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:38.272+0000 7f124a806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:38.341 INFO:tasks.rados.rados.0.smithi055.stdout:169: done (2 left) 2023-11-28T14:32:38.341 INFO:tasks.rados.rados.0.smithi055.stdout:170: done (1 left) 2023-11-28T14:32:38.341 INFO:tasks.rados.rados.0.smithi055.stdout:171: done (0 left) 2023-11-28T14:32:38.341 INFO:tasks.rados.rados.0.smithi055.stdout:172: delete oid 23 current snap is 13 2023-11-28T14:32:38.390 INFO:tasks.rados.rados.0.smithi055.stdout:172: done (0 left) 2023-11-28T14:32:38.390 INFO:tasks.rados.rados.0.smithi055.stdout:173: delete oid 36 current snap is 13 2023-11-28T14:32:38.435 INFO:tasks.rados.rados.0.smithi055.stdout:173: done (0 left) 2023-11-28T14:32:38.435 INFO:tasks.rados.rados.0.smithi055.stdout:174: write oid 41 current snap is 13 2023-11-28T14:32:38.435 INFO:tasks.rados.rados.0.smithi055.stdout:174: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2023-11-28T14:32:38.440 INFO:tasks.rados.rados.0.smithi055.stdout:174: writing smithi05583006-41 from 598349 to 1078934 tid 1 2023-11-28T14:32:38.446 INFO:tasks.rados.rados.0.smithi055.stdout:174: writing smithi05583006-41 from 1544985 to 2217688 tid 2 2023-11-28T14:32:38.453 INFO:tasks.rados.rados.0.smithi055.stdout:174: writing smithi05583006-41 from 2905319 to 3387542 tid 3 2023-11-28T14:32:38.453 INFO:tasks.rados.rados.0.smithi055.stdout:175: write oid 20 current snap is 13 2023-11-28T14:32:38.453 INFO:tasks.rados.rados.0.smithi055.stdout:175: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2023-11-28T14:32:38.460 INFO:tasks.rados.rados.0.smithi055.stdout:175: writing smithi05583006-20 from 615156 to 1170990 tid 1 2023-11-28T14:32:38.465 INFO:tasks.rados.rados.0.smithi055.stdout:175: writing smithi05583006-20 from 1603467 to 2019828 tid 2 2023-11-28T14:32:38.473 INFO:tasks.rados.rados.0.smithi055.stdout:175: writing smithi05583006-20 from 2816389 to 3509884 tid 3 2023-11-28T14:32:38.473 INFO:tasks.rados.rados.0.smithi055.stdout:176: snap_remove snap 12 2023-11-28T14:32:38.749 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:32:38.749 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:32:38.749 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:32:38.864 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:38.858+0000 7fe3d755e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:38.867 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:38.865+0000 7fe3d755e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:39.051 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:39.052 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:32:39.063 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2023-11-28T14:32:38.465078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115700191,"num_objects":75,"num_object_clones":30,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":384,"num_read_kb":153932,"num_write":522,"num_write_kb":112666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":514,"ondisk_log_size":514,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":320200,"kb_used_data":211800,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377167160,"statfs":{"total":386547056640,"available":386219171840,"internally_reserved":0,"allocated":216883200,"data_stored":229423183,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":890771,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3410,"num_write":10,"num_write_kb":3552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000689"},"pg_stats":[{"pgid":"3.f","version":"41'97","reported_seq":153,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T14:32:36.373714+0000","last_change":"2023-11-28T14:32:36.536032+0000","last_active":"2023-11-28T14:32:36.373714+0000","last_peered":"2023-11-28T14:32:36.373714+0000","last_clean":"2023-11-28T14:32:36.373714+0000","last_became_active":"2023-11-28T14:32:09.321992+0000","last_became_peered":"2023-11-28T14:32:09.321992+0000","last_unstale":"2023-11-28T14:32:36.373714+0000","last_undegraded":"2023-11-28T14:32:36.373714+0000","last_fullsized":"2023-11-28T14:32:36.373714+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563150,"num_objects":14,"num_object_clones":8,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":50,"num_read_kb":24715,"num_write":81,"num_write_kb":23846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"19'3","reported_seq":59,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.362140+0000","last_change":"2023-11-28T14:32:33.375979+0000","last_active":"2023-11-28T14:32:33.362140+0000","last_peered":"2023-11-28T14:32:33.362140+0000","last_clean":"2023-11-28T14:32:33.362140+0000","last_became_active":"2023-11-28T14:32:09.308848+0000","last_became_peered":"2023-11-28T14:32:09.308848+0000","last_unstale":"2023-11-28T14:32:33.362140+0000","last_undegraded":"2023-11-28T14:32:33.362140+0000","last_fullsized":"2023-11-28T14:32:33.362140+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":6433,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'15","reported_seq":79,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.371633+0000","last_change":"2023-11-28T14:32:33.387406+0000","last_active":"2023-11-28T14:32:33.371633+0000","last_peered":"2023-11-28T14:32:33.371633+0000","last_clean":"2023-11-28T14:32:33.371633+0000","last_became_active":"2023-11-28T14:32:09.337560+0000","last_became_peered":"2023-11-28T14:32:09.337560+0000","last_unstale":"2023-11-28T14:32:33.371633+0000","last_undegraded":"2023-11-28T14:32:33.371633+0000","last_fullsized":"2023-11-28T14:32:33.371633+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7848770,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":24013,"num_write":16,"num_write_kb":4334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"29'26","reported_seq":91,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T14:32:36.411536+0000","last_change":"2023-11-28T14:32:36.374386+0000","last_active":"2023-11-28T14:32:36.411536+0000","last_peered":"2023-11-28T14:32:36.411536+0000","last_clean":"2023-11-28T14:32:36.411536+0000","last_became_active":"2023-11-28T14:32:09.344782+0000","last_became_peered":"2023-11-28T14:32:09.344782+0000","last_unstale":"2023-11-28T14:32:36.411536+0000","last_undegraded":"2023-11-28T14:32:36.411536+0000","last_fullsized":"2023-11-28T14:32:36.411536+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11058874,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":13320,"num_write":26,"num_write_kb":7877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'23","reported_seq":81,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.314490+0000","last_change":"2023-11-28T14:32:33.356839+0000","last_active":"2023-11-28T14:32:33.314490+0000","last_peered":"2023-11-28T14:32:33.314490+0000","last_clean":"2023-11-28T14:32:33.314490+0000","last_became_active":"2023-11-28T14:32:09.326269+0000","last_became_peered":"2023-11-28T14:32:09.326269+0000","last_unstale":"2023-11-28T14:32:33.314490+0000","last_undegraded":"2023-11-28T14:32:33.314490+0000","last_fullsized":"2023-11-28T14:32:33.314490+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8470629,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":7024,"num_write":26,"num_write_kb":7179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"41'18","reported_seq":79,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T14:32:36.358665+0000","last_change":"2023-11-28T14:32:36.486462+0000","last_active":"2023-11-28T14:32:36.358665+0000","last_peered":"2023-11-28T14:32:36.358665+0000","last_clean":"2023-11-28T14:32:36.358665+0000","last_became_active":"2023-11-28T14:32:09.308846+0000","last_became_peered":"2023-11-28T14:32:09.308846+0000","last_unstale":"2023-11-28T14:32:36.358665+0000","last_undegraded":"2023-11-28T14:32:36.358665+0000","last_fullsized":"2023-11-28T14:32:36.358665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8084068,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":5567,"num_write":17,"num_write_kb":4623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"41'27","reported_seq":86,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T14:32:36.364766+0000","last_change":"2023-11-28T14:32:36.460357+0000","last_active":"2023-11-28T14:32:36.364766+0000","last_peered":"2023-11-28T14:32:36.364766+0000","last_clean":"2023-11-28T14:32:36.364766+0000","last_became_active":"2023-11-28T14:32:09.316660+0000","last_became_peered":"2023-11-28T14:32:09.316660+0000","last_unstale":"2023-11-28T14:32:36.364766+0000","last_undegraded":"2023-11-28T14:32:36.364766+0000","last_fullsized":"2023-11-28T14:32:36.364766+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226911,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":11,"num_read_kb":7663,"num_write":23,"num_write_kb":6709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"37'20","reported_seq":76,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.384090+0000","last_change":"2023-11-28T14:32:33.389805+0000","last_active":"2023-11-28T14:32:33.384090+0000","last_peered":"2023-11-28T14:32:33.384090+0000","last_clean":"2023-11-28T14:32:33.384090+0000","last_became_active":"2023-11-28T14:32:09.358025+0000","last_became_peered":"2023-11-28T14:32:09.358025+0000","last_unstale":"2023-11-28T14:32:33.384090+0000","last_undegraded":"2023-11-28T14:32:33.384090+0000","last_fullsized":"2023-11-28T14:32:33.384090+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":3595,"num_write":21,"num_write_kb":5738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":182,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.333106+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:32:33.333106+0000","last_peered":"2023-11-28T14:32:33.333106+0000","last_clean":"2023-11-28T14:32:33.333106+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:32:33.333106+0000","last_undegraded":"2023-11-28T14:32:33.333106+0000","last_fullsized":"2023-11-28T14:32:33.333106+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.376364+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:32:33.376364+0000","last_peered":"2023-11-28T14:32:33.376364+0000","last_clean":"2023-11-28T14:32:33.376364+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:32:33.376364+0000","last_undegraded":"2023-11-28T14:32:33.376364+0000","last_fullsized":"2023-11-28T14:32:33.376364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"37'69","reported_seq":134,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.375464+0000","last_change":"2023-11-28T14:32:33.384906+0000","last_active":"2023-11-28T14:32:33.375464+0000","last_peered":"2023-11-28T14:32:33.375464+0000","last_clean":"2023-11-28T14:32:33.375464+0000","last_became_active":"2023-11-28T14:32:09.335805+0000","last_became_peered":"2023-11-28T14:32:09.335805+0000","last_unstale":"2023-11-28T14:32:33.375464+0000","last_undegraded":"2023-11-28T14:32:33.375464+0000","last_fullsized":"2023-11-28T14:32:33.375464+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19983292,"num_objects":12,"num_object_clones":5,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":25954,"num_write":65,"num_write_kb":18061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T14:32:36.358701+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:32:36.358701+0000","last_peered":"2023-11-28T14:32:36.358701+0000","last_clean":"2023-11-28T14:32:36.358701+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:32:36.358701+0000","last_undegraded":"2023-11-28T14:32:36.358701+0000","last_fullsized":"2023-11-28T14:32:36.358701+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"38'28","reported_seq":80,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.317455+0000","last_change":"2023-11-28T14:32:33.369106+0000","last_active":"2023-11-28T14:32:33.317455+0000","last_peered":"2023-11-28T14:32:33.317455+0000","last_clean":"2023-11-28T14:32:33.317455+0000","last_became_active":"2023-11-28T14:32:09.350412+0000","last_became_peered":"2023-11-28T14:32:09.350412+0000","last_unstale":"2023-11-28T14:32:33.317455+0000","last_undegraded":"2023-11-28T14:32:33.317455+0000","last_fullsized":"2023-11-28T14:32:33.317455+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5680700,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":18,"num_read_kb":8137,"num_write":21,"num_write_kb":6458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.380013+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:32:33.380013+0000","last_peered":"2023-11-28T14:32:33.380013+0000","last_clean":"2023-11-28T14:32:33.380013+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:32:33.380013+0000","last_undegraded":"2023-11-28T14:32:33.380013+0000","last_fullsized":"2023-11-28T14:32:33.380013+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"37'19","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.331402+0000","last_change":"2023-11-28T14:32:33.337548+0000","last_active":"2023-11-28T14:32:33.331402+0000","last_peered":"2023-11-28T14:32:33.331402+0000","last_clean":"2023-11-28T14:32:33.331402+0000","last_became_active":"2023-11-28T14:32:09.343664+0000","last_became_peered":"2023-11-28T14:32:09.343664+0000","last_unstale":"2023-11-28T14:32:33.331402+0000","last_undegraded":"2023-11-28T14:32:33.331402+0000","last_fullsized":"2023-11-28T14:32:33.331402+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3546244,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":14,"num_write_kb":3792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T14:32:36.363761+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:32:36.363761+0000","last_peered":"2023-11-28T14:32:36.363761+0000","last_clean":"2023-11-28T14:32:36.363761+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:32:36.363761+0000","last_undegraded":"2023-11-28T14:32:36.363761+0000","last_fullsized":"2023-11-28T14:32:36.363761+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"19'3","reported_seq":56,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.363228+0000","last_change":"2023-11-28T14:32:33.376550+0000","last_active":"2023-11-28T14:32:33.363228+0000","last_peered":"2023-11-28T14:32:33.363228+0000","last_clean":"2023-11-28T14:32:33.363228+0000","last_became_active":"2023-11-28T14:32:09.311653+0000","last_became_peered":"2023-11-28T14:32:09.311653+0000","last_unstale":"2023-11-28T14:32:33.363228+0000","last_undegraded":"2023-11-28T14:32:33.363228+0000","last_fullsized":"2023-11-28T14:32:33.363228+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016807,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"38'23","reported_seq":72,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.384070+0000","last_change":"2023-11-28T14:32:33.412717+0000","last_active":"2023-11-28T14:32:33.384070+0000","last_peered":"2023-11-28T14:32:33.384070+0000","last_clean":"2023-11-28T14:32:33.384070+0000","last_became_active":"2023-11-28T14:32:09.316944+0000","last_became_peered":"2023-11-28T14:32:09.316944+0000","last_unstale":"2023-11-28T14:32:33.384070+0000","last_undegraded":"2023-11-28T14:32:33.384070+0000","last_fullsized":"2023-11-28T14:32:33.384070+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3630279,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":21,"num_write_kb":6458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.362356+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:32:33.362356+0000","last_peered":"2023-11-28T14:32:33.362356+0000","last_clean":"2023-11-28T14:32:33.362356+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:32:33.362356+0000","last_undegraded":"2023-11-28T14:32:33.362356+0000","last_fullsized":"2023-11-28T14:32:33.362356+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"23'13","reported_seq":69,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.364025+0000","last_change":"2023-11-28T14:32:33.374321+0000","last_active":"2023-11-28T14:32:33.364025+0000","last_peered":"2023-11-28T14:32:33.364025+0000","last_clean":"2023-11-28T14:32:33.364025+0000","last_became_active":"2023-11-28T14:32:09.309510+0000","last_became_peered":"2023-11-28T14:32:09.309510+0000","last_unstale":"2023-11-28T14:32:33.364025+0000","last_undegraded":"2023-11-28T14:32:33.364025+0000","last_fullsized":"2023-11-28T14:32:33.364025+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2386561,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":7922,"num_write":12,"num_write_kb":3313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.327556+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:32:33.327556+0000","last_peered":"2023-11-28T14:32:33.327556+0000","last_clean":"2023-11-28T14:32:33.327556+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:32:33.327556+0000","last_undegraded":"2023-11-28T14:32:33.327556+0000","last_fullsized":"2023-11-28T14:32:33.327556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"23'27","reported_seq":83,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.334983+0000","last_change":"2023-11-28T14:32:33.359622+0000","last_active":"2023-11-28T14:32:33.334983+0000","last_peered":"2023-11-28T14:32:33.334983+0000","last_clean":"2023-11-28T14:32:33.334983+0000","last_became_active":"2023-11-28T14:32:09.356991+0000","last_became_peered":"2023-11-28T14:32:09.356991+0000","last_unstale":"2023-11-28T14:32:33.334983+0000","last_undegraded":"2023-11-28T14:32:33.334983+0000","last_fullsized":"2023-11-28T14:32:33.334983+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151296,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":9602,"num_write":26,"num_write_kb":8152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.369626+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:32:33.369626+0000","last_peered":"2023-11-28T14:32:33.369626+0000","last_clean":"2023-11-28T14:32:33.369626+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:32:33.369626+0000","last_undegraded":"2023-11-28T14:32:33.369626+0000","last_fullsized":"2023-11-28T14:32:33.369626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"19'6","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.371330+0000","last_change":"2023-11-28T14:32:33.376229+0000","last_active":"2023-11-28T14:32:33.371330+0000","last_peered":"2023-11-28T14:32:33.371330+0000","last_clean":"2023-11-28T14:32:33.371330+0000","last_became_active":"2023-11-28T14:32:09.337548+0000","last_became_peered":"2023-11-28T14:32:09.337548+0000","last_unstale":"2023-11-28T14:32:33.371330+0000","last_undegraded":"2023-11-28T14:32:33.371330+0000","last_fullsized":"2023-11-28T14:32:33.371330+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":9887,"num_write":8,"num_write_kb":2059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T14:32:33.379835+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:32:33.379835+0000","last_peered":"2023-11-28T14:32:33.379835+0000","last_clean":"2023-11-28T14:32:33.379835+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:32:33.379835+0000","last_undegraded":"2023-11-28T14:32:33.379835+0000","last_fullsized":"2023-11-28T14:32:33.379835+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":115109823,"num_objects":73,"num_object_clones":30,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":281,"num_read_kb":153841,"num_write":385,"num_write_kb":110772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":417,"ondisk_log_size":417,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78288,"kb_used_data":50360,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94293552,"statfs":{"total":96636764160,"available":96556597248,"internally_reserved":0,"allocated":51568640,"data_stored":51534951,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705700,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91008,"kb_used_data":64184,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94280832,"statfs":{"total":96636764160,"available":96543571968,"internally_reserved":0,"allocated":65724416,"data_stored":70163769,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":63480,"kb_used_data":36656,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94308360,"statfs":{"total":96636764160,"available":96571760640,"internally_reserved":0,"allocated":37535744,"data_stored":40849677,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705700,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87424,"kb_used_data":60600,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94284416,"statfs":{"total":96636764160,"available":96547241984,"internally_reserved":0,"allocated":62054400,"data_stored":66874786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:32:39.350 INFO:tasks.rados.rados.0.smithi055.stdout:176: done (2 left) 2023-11-28T14:32:39.350 INFO:tasks.rados.rados.0.smithi055.stdout:177: delete oid 30 current snap is 13 2023-11-28T14:32:39.350 INFO:tasks.rados.rados.0.smithi055.stdout:174: finishing write tid 1 to smithi05583006-41 2023-11-28T14:32:39.350 INFO:tasks.rados.rados.0.smithi055.stdout:174: finishing write tid 2 to smithi05583006-41 2023-11-28T14:32:39.350 INFO:tasks.rados.rados.0.smithi055.stdout:175: finishing write tid 1 to smithi05583006-20 2023-11-28T14:32:39.350 INFO:tasks.rados.rados.0.smithi055.stdout:174: finishing write tid 3 to smithi05583006-41 2023-11-28T14:32:39.350 INFO:tasks.rados.rados.0.smithi055.stdout:175: finishing write tid 2 to smithi05583006-20 2023-11-28T14:32:39.350 INFO:tasks.rados.rados.0.smithi055.stdout:174: finishing write tid 4 to smithi05583006-41 2023-11-28T14:32:39.350 INFO:tasks.rados.rados.0.smithi055.stdout:174: finishing write tid 5 to smithi05583006-41 2023-11-28T14:32:39.351 INFO:tasks.rados.rados.0.smithi055.stdout:174: oid 41 updating version 0 to 99 2023-11-28T14:32:39.351 INFO:tasks.rados.rados.0.smithi055.stdout:174: oid 41 updating version 99 to 101 2023-11-28T14:32:39.351 INFO:tasks.rados.rados.0.smithi055.stdout:174: oid 41 updating version 101 to 102 2023-11-28T14:32:39.351 INFO:tasks.rados.rados.0.smithi055.stdout:174: oid 41 version 102 is already newer than 100 2023-11-28T14:32:39.351 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 41 v 102 (ObjNum 107 snap 13 seq_num 107) dirty exists 2023-11-28T14:32:39.351 INFO:tasks.rados.rados.0.smithi055.stdout:174: left oid 41 (ObjNum 107 snap 13 seq_num 107) 2023-11-28T14:32:39.351 INFO:tasks.rados.rados.0.smithi055.stdout:175: finishing write tid 3 to smithi05583006-20 2023-11-28T14:32:39.351 INFO:tasks.rados.rados.0.smithi055.stdout:175: finishing write tid 4 to smithi05583006-20 2023-11-28T14:32:39.351 INFO:tasks.rados.rados.0.smithi055.stdout:175: finishing write tid 5 to smithi05583006-20 2023-11-28T14:32:39.351 INFO:tasks.rados.rados.0.smithi055.stdout:175: oid 20 updating version 0 to 9 2023-11-28T14:32:39.351 INFO:tasks.rados.rados.0.smithi055.stdout:175: oid 20 updating version 9 to 10 2023-11-28T14:32:39.351 INFO:tasks.rados.rados.0.smithi055.stdout:175: oid 20 updating version 10 to 11 2023-11-28T14:32:39.352 INFO:tasks.rados.rados.0.smithi055.stdout:175: oid 20 version 11 is already newer than 8 2023-11-28T14:32:39.352 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 20 v 11 (ObjNum 108 snap 13 seq_num 108) dirty exists 2023-11-28T14:32:39.352 INFO:tasks.rados.rados.0.smithi055.stdout:175: left oid 20 (ObjNum 108 snap 13 seq_num 108) 2023-11-28T14:32:39.378 INFO:tasks.rados.rados.0.smithi055.stdout:174: done (2 left) 2023-11-28T14:32:39.378 INFO:tasks.rados.rados.0.smithi055.stdout:175: done (1 left) 2023-11-28T14:32:39.378 INFO:tasks.rados.rados.0.smithi055.stdout:177: done (0 left) 2023-11-28T14:32:39.378 INFO:tasks.rados.rados.0.smithi055.stdout:178: write (excl) oid 45 current snap is 13 2023-11-28T14:32:39.378 INFO:tasks.rados.rados.0.smithi055.stdout:178: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2023-11-28T14:32:39.385 INFO:tasks.rados.rados.0.smithi055.stdout:178: writing smithi05583006-45 from 648770 to 1355102 tid 1 2023-11-28T14:32:39.391 INFO:tasks.rados.rados.0.smithi055.stdout:178: writing smithi05583006-45 from 2120431 to 2540461 tid 2 2023-11-28T14:32:39.398 INFO:tasks.rados.rados.0.smithi055.stdout:178: writing smithi05583006-45 from 3271235 to 3703627 tid 3 2023-11-28T14:32:39.398 INFO:tasks.rados.rados.0.smithi055.stdout:179: snap_remove snap 11 2023-11-28T14:32:40.349 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:32:40.355 INFO:tasks.rados.rados.0.smithi055.stdout:179: done (1 left) 2023-11-28T14:32:40.355 INFO:tasks.rados.rados.0.smithi055.stdout:180: snap_create 2023-11-28T14:32:40.355 INFO:tasks.rados.rados.0.smithi055.stdout:178: finishing write tid 1 to smithi05583006-45 2023-11-28T14:32:40.356 INFO:tasks.rados.rados.0.smithi055.stdout:178: finishing write tid 2 to smithi05583006-45 2023-11-28T14:32:40.356 INFO:tasks.rados.rados.0.smithi055.stdout:178: finishing write tid 3 to smithi05583006-45 2023-11-28T14:32:40.356 INFO:tasks.rados.rados.0.smithi055.stdout:178: finishing write tid 4 to smithi05583006-45 2023-11-28T14:32:40.356 INFO:tasks.rados.rados.0.smithi055.stdout:178: finishing write tid 5 to smithi05583006-45 2023-11-28T14:32:40.356 INFO:tasks.rados.rados.0.smithi055.stdout:178: oid 45 updating version 0 to 34 2023-11-28T14:32:40.356 INFO:tasks.rados.rados.0.smithi055.stdout:178: oid 45 updating version 34 to 37 2023-11-28T14:32:40.356 INFO:tasks.rados.rados.0.smithi055.stdout:178: oid 45 version 37 is already newer than 36 2023-11-28T14:32:40.356 INFO:tasks.rados.rados.0.smithi055.stdout:178: oid 45 version 37 is already newer than 35 2023-11-28T14:32:40.356 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 45 v 37 (ObjNum 110 snap 13 seq_num 110) dirty exists 2023-11-28T14:32:40.356 INFO:tasks.rados.rados.0.smithi055.stdout:178: left oid 45 (ObjNum 110 snap 13 seq_num 110) 2023-11-28T14:32:40.475 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:40.472+0000 7f320e300640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:40.478 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:40.476+0000 7f320e300640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:41.365 INFO:tasks.rados.rados.0.smithi055.stdout:178: done (1 left) 2023-11-28T14:32:41.366 INFO:tasks.rados.rados.0.smithi055.stdout:180: done (0 left) 2023-11-28T14:32:41.366 INFO:tasks.rados.rados.0.smithi055.stdout:181: delete oid 49 current snap is 14 2023-11-28T14:32:41.493 INFO:tasks.rados.rados.0.smithi055.stdout:181: done (0 left) 2023-11-28T14:32:41.493 INFO:tasks.rados.rados.0.smithi055.stdout:182: write (excl) oid 46 current snap is 14 2023-11-28T14:32:41.493 INFO:tasks.rados.rados.0.smithi055.stdout:182: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2023-11-28T14:32:41.499 INFO:tasks.rados.rados.0.smithi055.stdout:182: writing smithi05583006-46 from 682384 to 1139214 tid 1 2023-11-28T14:32:41.504 INFO:tasks.rados.rados.0.smithi055.stdout:182: writing smithi05583006-46 from 1721042 to 2144741 tid 2 2023-11-28T14:32:41.510 INFO:tasks.rados.rados.0.smithi055.stdout:182: writing smithi05583006-46 from 2809728 to 3264664 tid 3 2023-11-28T14:32:41.514 INFO:tasks.rados.rados.0.smithi055.stdout:182: writing smithi05583006-46 from 3731357 to 3882384 tid 4 2023-11-28T14:32:41.514 INFO:tasks.rados.rados.0.smithi055.stdout:183: snap_remove snap 13 2023-11-28T14:32:42.369 INFO:tasks.rados.rados.0.smithi055.stdout:183: done (1 left) 2023-11-28T14:32:42.369 INFO:tasks.rados.rados.0.smithi055.stdout:184: snap_create 2023-11-28T14:32:42.369 INFO:tasks.rados.rados.0.smithi055.stdout:182: finishing write tid 1 to smithi05583006-46 2023-11-28T14:32:42.369 INFO:tasks.rados.rados.0.smithi055.stdout:182: finishing write tid 2 to smithi05583006-46 2023-11-28T14:32:42.369 INFO:tasks.rados.rados.0.smithi055.stdout:182: finishing write tid 3 to smithi05583006-46 2023-11-28T14:32:42.369 INFO:tasks.rados.rados.0.smithi055.stdout:182: finishing write tid 4 to smithi05583006-46 2023-11-28T14:32:42.369 INFO:tasks.rados.rados.0.smithi055.stdout:182: finishing write tid 5 to smithi05583006-46 2023-11-28T14:32:42.369 INFO:tasks.rados.rados.0.smithi055.stdout:182: finishing write tid 6 to smithi05583006-46 2023-11-28T14:32:42.370 INFO:tasks.rados.rados.0.smithi055.stdout:182: oid 46 updating version 0 to 85 2023-11-28T14:32:42.370 INFO:tasks.rados.rados.0.smithi055.stdout:182: oid 46 version 85 is already newer than 82 2023-11-28T14:32:42.370 INFO:tasks.rados.rados.0.smithi055.stdout:182: oid 46 version 85 is already newer than 85 2023-11-28T14:32:42.370 INFO:tasks.rados.rados.0.smithi055.stdout:182: oid 46 version 85 is already newer than 81 2023-11-28T14:32:42.370 INFO:tasks.rados.rados.0.smithi055.stdout:182: oid 46 version 85 is already newer than 83 2023-11-28T14:32:42.370 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 46 v 85 (ObjNum 112 snap 14 seq_num 112) dirty exists 2023-11-28T14:32:42.370 INFO:tasks.rados.rados.0.smithi055.stdout:182: left oid 46 (ObjNum 112 snap 14 seq_num 112) 2023-11-28T14:32:42.552 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:32:42.676 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:42.673+0000 7fb25af1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:42.679 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:42.678+0000 7fb25af1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:43.373 INFO:tasks.rados.rados.0.smithi055.stdout:182: done (1 left) 2023-11-28T14:32:43.373 INFO:tasks.rados.rados.0.smithi055.stdout:184: done (0 left) 2023-11-28T14:32:43.373 INFO:tasks.rados.rados.0.smithi055.stdout:185: snap_create 2023-11-28T14:32:44.065 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:32:44.066 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:32:44.066 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:32:44.193 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:44.189+0000 7f2dc519e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:44.196 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:44.195+0000 7f2dc519e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:44.378 INFO:tasks.rados.rados.0.smithi055.stdout:185: done (0 left) 2023-11-28T14:32:44.379 INFO:tasks.rados.rados.0.smithi055.stdout:186: read oid 2 snap -1 2023-11-28T14:32:44.379 INFO:tasks.rados.rados.0.smithi055.stdout:186: expect (ObjNum 102 snap 11 seq_num 102) 2023-11-28T14:32:44.379 INFO:tasks.rados.rados.0.smithi055.stdout:187: snap_create 2023-11-28T14:32:44.388 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:44.389 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:32:44.405 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2023-11-28T14:32:44.376261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109649803,"num_objects":84,"num_object_clones":39,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":401,"num_read_kb":159848,"num_write":553,"num_write_kb":122056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":592,"ondisk_log_size":592,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":297644,"kb_used_data":190348,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377189716,"statfs":{"total":386547056640,"available":386242269184,"internally_reserved":0,"allocated":194916352,"data_stored":195546525,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3428562,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":7,"num_read_kb":4,"num_write":19,"num_write_kb":5254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.024041"},"pg_stats":[{"pgid":"3.f","version":"45'106","reported_seq":168,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-11-28T14:32:41.422559+0000","last_change":"2023-11-28T14:32:41.422559+0000","last_active":"2023-11-28T14:32:41.422559+0000","last_peered":"2023-11-28T14:32:41.422559+0000","last_clean":"2023-11-28T14:32:41.422559+0000","last_became_active":"2023-11-28T14:32:09.321992+0000","last_became_peered":"2023-11-28T14:32:09.321992+0000","last_unstale":"2023-11-28T14:32:41.422559+0000","last_undegraded":"2023-11-28T14:32:41.422559+0000","last_fullsized":"2023-11-28T14:32:41.422559+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17865676,"num_objects":15,"num_object_clones":9,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":51,"num_read_kb":24716,"num_write":86,"num_write_kb":25444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"45'10","reported_seq":89,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-28T14:32:43.385930+0000","last_change":"2023-11-28T14:32:42.767328+0000","last_active":"2023-11-28T14:32:43.385930+0000","last_peered":"2023-11-28T14:32:43.385930+0000","last_clean":"2023-11-28T14:32:43.385930+0000","last_became_active":"2023-11-28T14:32:09.308848+0000","last_became_peered":"2023-11-28T14:32:09.308848+0000","last_unstale":"2023-11-28T14:32:43.385930+0000","last_undegraded":"2023-11-28T14:32:43.385930+0000","last_fullsized":"2023-11-28T14:32:43.385930+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3747928,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":6434,"num_write":9,"num_write_kb":2690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'15","reported_seq":100,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T14:32:42.774370+0000","last_change":"2023-11-28T14:32:42.780170+0000","last_active":"2023-11-28T14:32:42.774370+0000","last_peered":"2023-11-28T14:32:42.774370+0000","last_clean":"2023-11-28T14:32:42.774370+0000","last_became_active":"2023-11-28T14:32:09.337560+0000","last_became_peered":"2023-11-28T14:32:09.337560+0000","last_unstale":"2023-11-28T14:32:42.774370+0000","last_undegraded":"2023-11-28T14:32:42.774370+0000","last_fullsized":"2023-11-28T14:32:42.774370+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7848770,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":24013,"num_write":16,"num_write_kb":4334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"45'32","reported_seq":102,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-11-28T14:32:41.420667+0000","last_change":"2023-11-28T14:32:41.420667+0000","last_active":"2023-11-28T14:32:41.420667+0000","last_peered":"2023-11-28T14:32:41.420667+0000","last_clean":"2023-11-28T14:32:41.420667+0000","last_became_active":"2023-11-28T14:32:09.344782+0000","last_became_peered":"2023-11-28T14:32:09.344782+0000","last_unstale":"2023-11-28T14:32:41.420667+0000","last_undegraded":"2023-11-28T14:32:41.420667+0000","last_fullsized":"2023-11-28T14:32:41.420667+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8470629,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":22,"num_read_kb":13320,"num_write":26,"num_write_kb":7877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'23","reported_seq":102,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T14:32:42.379861+0000","last_change":"2023-11-28T14:32:42.391016+0000","last_active":"2023-11-28T14:32:42.379861+0000","last_peered":"2023-11-28T14:32:42.379861+0000","last_clean":"2023-11-28T14:32:42.379861+0000","last_became_active":"2023-11-28T14:32:09.326269+0000","last_became_peered":"2023-11-28T14:32:09.326269+0000","last_unstale":"2023-11-28T14:32:42.379861+0000","last_undegraded":"2023-11-28T14:32:42.379861+0000","last_fullsized":"2023-11-28T14:32:42.379861+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8470629,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":7024,"num_write":26,"num_write_kb":7179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"41'18","reported_seq":89,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-11-28T14:32:41.421869+0000","last_change":"2023-11-28T14:32:41.356552+0000","last_active":"2023-11-28T14:32:41.421869+0000","last_peered":"2023-11-28T14:32:41.421869+0000","last_clean":"2023-11-28T14:32:41.421869+0000","last_became_active":"2023-11-28T14:32:09.308846+0000","last_became_peered":"2023-11-28T14:32:09.308846+0000","last_unstale":"2023-11-28T14:32:41.421869+0000","last_undegraded":"2023-11-28T14:32:41.421869+0000","last_fullsized":"2023-11-28T14:32:41.421869+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8084068,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":5567,"num_write":17,"num_write_kb":4623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"41'27","reported_seq":96,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-11-28T14:32:41.416620+0000","last_change":"2023-11-28T14:32:41.351527+0000","last_active":"2023-11-28T14:32:41.416620+0000","last_peered":"2023-11-28T14:32:41.416620+0000","last_clean":"2023-11-28T14:32:41.416620+0000","last_became_active":"2023-11-28T14:32:09.316660+0000","last_became_peered":"2023-11-28T14:32:09.316660+0000","last_unstale":"2023-11-28T14:32:41.416620+0000","last_undegraded":"2023-11-28T14:32:41.416620+0000","last_fullsized":"2023-11-28T14:32:41.416620+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226911,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":11,"num_read_kb":7663,"num_write":23,"num_write_kb":6709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"45'29","reported_seq":98,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T14:32:42.773579+0000","last_change":"2023-11-28T14:32:42.780524+0000","last_active":"2023-11-28T14:32:42.773579+0000","last_peered":"2023-11-28T14:32:42.773579+0000","last_clean":"2023-11-28T14:32:42.773579+0000","last_became_active":"2023-11-28T14:32:09.358025+0000","last_became_peered":"2023-11-28T14:32:09.358025+0000","last_unstale":"2023-11-28T14:32:42.773579+0000","last_undegraded":"2023-11-28T14:32:42.773579+0000","last_fullsized":"2023-11-28T14:32:42.773579+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5680700,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":15,"num_read_kb":3595,"num_write":21,"num_write_kb":5738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":198,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T14:32:42.378674+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:32:42.378674+0000","last_peered":"2023-11-28T14:32:42.378674+0000","last_clean":"2023-11-28T14:32:42.378674+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:32:42.378674+0000","last_undegraded":"2023-11-28T14:32:42.378674+0000","last_fullsized":"2023-11-28T14:32:42.378674+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":61,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T14:32:42.769735+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:32:42.769735+0000","last_peered":"2023-11-28T14:32:42.769735+0000","last_clean":"2023-11-28T14:32:42.769735+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:32:42.769735+0000","last_undegraded":"2023-11-28T14:32:42.769735+0000","last_fullsized":"2023-11-28T14:32:42.769735+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"47'87","reported_seq":167,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T14:32:42.773765+0000","last_change":"2023-11-28T14:32:42.807487+0000","last_active":"2023-11-28T14:32:42.773765+0000","last_peered":"2023-11-28T14:32:42.773765+0000","last_clean":"2023-11-28T14:32:42.773765+0000","last_became_active":"2023-11-28T14:32:09.335805+0000","last_became_peered":"2023-11-28T14:32:09.335805+0000","last_unstale":"2023-11-28T14:32:42.773765+0000","last_undegraded":"2023-11-28T14:32:42.773765+0000","last_fullsized":"2023-11-28T14:32:42.773765+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19008519,"num_objects":14,"num_object_clones":7,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":57,"num_read_kb":25956,"num_write":76,"num_write_kb":21255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":59,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-11-28T14:32:41.419049+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:32:41.419049+0000","last_peered":"2023-11-28T14:32:41.419049+0000","last_clean":"2023-11-28T14:32:41.419049+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:32:41.419049+0000","last_undegraded":"2023-11-28T14:32:41.419049+0000","last_fullsized":"2023-11-28T14:32:41.419049+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"45'39","reported_seq":107,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T14:32:42.385877+0000","last_change":"2023-11-28T14:32:42.393199+0000","last_active":"2023-11-28T14:32:42.385877+0000","last_peered":"2023-11-28T14:32:42.385877+0000","last_clean":"2023-11-28T14:32:42.385877+0000","last_became_active":"2023-11-28T14:32:09.350412+0000","last_became_peered":"2023-11-28T14:32:09.350412+0000","last_unstale":"2023-11-28T14:32:42.385877+0000","last_undegraded":"2023-11-28T14:32:42.385877+0000","last_fullsized":"2023-11-28T14:32:42.385877+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6000033,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":20,"num_read_kb":8138,"num_write":26,"num_write_kb":7982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T14:32:42.772335+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:32:42.772335+0000","last_peered":"2023-11-28T14:32:42.772335+0000","last_clean":"2023-11-28T14:32:42.772335+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:32:42.772335+0000","last_undegraded":"2023-11-28T14:32:42.772335+0000","last_fullsized":"2023-11-28T14:32:42.772335+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"41'21","reported_seq":87,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T14:32:42.385396+0000","last_change":"2023-11-28T14:32:42.390553+0000","last_active":"2023-11-28T14:32:42.385396+0000","last_peered":"2023-11-28T14:32:42.385396+0000","last_clean":"2023-11-28T14:32:42.385396+0000","last_became_active":"2023-11-28T14:32:09.343664+0000","last_became_peered":"2023-11-28T14:32:09.343664+0000","last_unstale":"2023-11-28T14:32:42.385396+0000","last_undegraded":"2023-11-28T14:32:42.385396+0000","last_fullsized":"2023-11-28T14:32:42.385396+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3546244,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":14,"num_write_kb":3792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":59,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-11-28T14:32:41.404216+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:32:41.404216+0000","last_peered":"2023-11-28T14:32:41.404216+0000","last_clean":"2023-11-28T14:32:41.404216+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:32:41.404216+0000","last_undegraded":"2023-11-28T14:32:41.404216+0000","last_fullsized":"2023-11-28T14:32:41.404216+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"19'3","reported_seq":84,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-28T14:32:43.386146+0000","last_change":"2023-11-28T14:32:42.772807+0000","last_active":"2023-11-28T14:32:43.386146+0000","last_peered":"2023-11-28T14:32:43.386146+0000","last_clean":"2023-11-28T14:32:43.386146+0000","last_became_active":"2023-11-28T14:32:09.311653+0000","last_became_peered":"2023-11-28T14:32:09.311653+0000","last_unstale":"2023-11-28T14:32:43.386146+0000","last_undegraded":"2023-11-28T14:32:43.386146+0000","last_fullsized":"2023-11-28T14:32:43.386146+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016807,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5911,"num_write":4,"num_write_kb":929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"41'26","reported_seq":93,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T14:32:42.774068+0000","last_change":"2023-11-28T14:32:42.786860+0000","last_active":"2023-11-28T14:32:42.774068+0000","last_peered":"2023-11-28T14:32:42.774068+0000","last_clean":"2023-11-28T14:32:42.774068+0000","last_became_active":"2023-11-28T14:32:09.316944+0000","last_became_peered":"2023-11-28T14:32:09.316944+0000","last_unstale":"2023-11-28T14:32:42.774068+0000","last_undegraded":"2023-11-28T14:32:42.774068+0000","last_fullsized":"2023-11-28T14:32:42.774068+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3630279,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":21,"num_write_kb":6458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":69,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-28T14:32:43.382801+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:32:43.382801+0000","last_peered":"2023-11-28T14:32:43.382801+0000","last_clean":"2023-11-28T14:32:43.382801+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:32:43.382801+0000","last_undegraded":"2023-11-28T14:32:43.382801+0000","last_fullsized":"2023-11-28T14:32:43.382801+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"23'13","reported_seq":94,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-28T14:32:43.383655+0000","last_change":"2023-11-28T14:32:42.764297+0000","last_active":"2023-11-28T14:32:43.383655+0000","last_peered":"2023-11-28T14:32:43.383655+0000","last_clean":"2023-11-28T14:32:43.383655+0000","last_became_active":"2023-11-28T14:32:09.309510+0000","last_became_peered":"2023-11-28T14:32:09.309510+0000","last_unstale":"2023-11-28T14:32:43.383655+0000","last_undegraded":"2023-11-28T14:32:43.383655+0000","last_fullsized":"2023-11-28T14:32:43.383655+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2386561,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":7922,"num_write":12,"num_write_kb":3313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":65,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T14:32:42.390085+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:32:42.390085+0000","last_peered":"2023-11-28T14:32:42.390085+0000","last_clean":"2023-11-28T14:32:42.390085+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:32:42.390085+0000","last_undegraded":"2023-11-28T14:32:42.390085+0000","last_fullsized":"2023-11-28T14:32:42.390085+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"47'31","reported_seq":105,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T14:32:42.385743+0000","last_change":"2023-11-28T14:32:42.413896+0000","last_active":"2023-11-28T14:32:42.385743+0000","last_peered":"2023-11-28T14:32:42.385743+0000","last_clean":"2023-11-28T14:32:42.385743+0000","last_became_active":"2023-11-28T14:32:09.356991+0000","last_became_peered":"2023-11-28T14:32:09.356991+0000","last_unstale":"2023-11-28T14:32:42.385743+0000","last_undegraded":"2023-11-28T14:32:42.385743+0000","last_fullsized":"2023-11-28T14:32:42.385743+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823543,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":15,"num_read_kb":9602,"num_write":26,"num_write_kb":8152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":69,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-28T14:32:43.384454+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:32:43.384454+0000","last_peered":"2023-11-28T14:32:43.384454+0000","last_clean":"2023-11-28T14:32:43.384454+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:32:43.384454+0000","last_undegraded":"2023-11-28T14:32:43.384454+0000","last_fullsized":"2023-11-28T14:32:43.384454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"45'15","reported_seq":96,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-11-28T14:32:43.387143+0000","last_change":"2023-11-28T14:32:42.766924+0000","last_active":"2023-11-28T14:32:43.387143+0000","last_peered":"2023-11-28T14:32:43.387143+0000","last_clean":"2023-11-28T14:32:43.387143+0000","last_became_active":"2023-11-28T14:32:09.337548+0000","last_became_peered":"2023-11-28T14:32:09.337548+0000","last_unstale":"2023-11-28T14:32:43.387143+0000","last_undegraded":"2023-11-28T14:32:43.387143+0000","last_fullsized":"2023-11-28T14:32:43.387143+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6252138,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":9888,"num_write":13,"num_write_kb":3687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T14:32:42.765616+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:32:42.765616+0000","last_peered":"2023-11-28T14:32:42.765616+0000","last_clean":"2023-11-28T14:32:42.765616+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:32:42.765616+0000","last_undegraded":"2023-11-28T14:32:42.765616+0000","last_fullsized":"2023-11-28T14:32:42.765616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":109059435,"num_objects":82,"num_object_clones":39,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":298,"num_read_kb":159757,"num_write":416,"num_write_kb":120162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":495,"ondisk_log_size":495,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77252,"kb_used_data":50428,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94294588,"statfs":{"total":96636764160,"available":96557658112,"internally_reserved":0,"allocated":51638272,"data_stored":51590640,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705710,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82740,"kb_used_data":55916,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94289100,"statfs":{"total":96636764160,"available":96552038400,"internally_reserved":0,"allocated":57257984,"data_stored":57165412,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":58484,"kb_used_data":31660,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94313356,"statfs":{"total":96636764160,"available":96576876544,"internally_reserved":0,"allocated":32419840,"data_stored":32593828,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705709,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79168,"kb_used_data":52344,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94292672,"statfs":{"total":96636764160,"available":96555696128,"internally_reserved":0,"allocated":53600256,"data_stored":54196645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:32:44.453 INFO:tasks.rados.rados.0.smithi055.stdout:186: expect (ObjNum 102 snap 11 seq_num 102) 2023-11-28T14:32:44.767 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:32:44.877 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:44.875+0000 7f94bfc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:44.880 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:44.879+0000 7f94bfc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:45.387 INFO:tasks.rados.rados.0.smithi055.stdout:186: done (1 left) 2023-11-28T14:32:45.387 INFO:tasks.rados.rados.0.smithi055.stdout:187: done (0 left) 2023-11-28T14:32:45.387 INFO:tasks.rados.rados.0.smithi055.stdout:188: delete oid 46 current snap is 17 2023-11-28T14:32:45.434 INFO:tasks.rados.rados.0.smithi055.stdout:188: done (0 left) 2023-11-28T14:32:45.434 INFO:tasks.rados.rados.0.smithi055.stdout:189: snap_remove snap 14 2023-11-28T14:32:46.401 INFO:tasks.rados.rados.0.smithi055.stdout:189: done (0 left) 2023-11-28T14:32:46.401 INFO:tasks.rados.rados.0.smithi055.stdout:190: write (excl) oid 40 current snap is 17 2023-11-28T14:32:46.401 INFO:tasks.rados.rados.0.smithi055.stdout:190: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2023-11-28T14:32:46.409 INFO:tasks.rados.rados.0.smithi055.stdout:190: writing smithi05583006-40 from 715998 to 1323326 tid 1 2023-11-28T14:32:46.415 INFO:tasks.rados.rados.0.smithi055.stdout:190: writing smithi05583006-40 from 1838006 to 2265374 tid 2 2023-11-28T14:32:46.423 INFO:tasks.rados.rados.0.smithi055.stdout:190: writing smithi05583006-40 from 2864574 to 3342054 tid 3 2023-11-28T14:32:46.427 INFO:tasks.rados.rados.0.smithi055.stdout:190: writing smithi05583006-40 from 3863835 to 3915998 tid 4 2023-11-28T14:32:46.427 INFO:tasks.rados.rados.0.smithi055.stdout:191: read oid 8 snap 16 2023-11-28T14:32:46.427 INFO:tasks.rados.rados.0.smithi055.stdout:191: expect deleted 2023-11-28T14:32:46.427 INFO:tasks.rados.rados.0.smithi055.stdout:192: read oid 19 snap 15 2023-11-28T14:32:46.427 INFO:tasks.rados.rados.0.smithi055.stdout:192: expect (ObjNum 18 snap 0 seq_num 18) 2023-11-28T14:32:46.427 INFO:tasks.rados.rados.0.smithi055.stdout:193: snap_create 2023-11-28T14:32:46.477 INFO:tasks.rados.rados.0.smithi055.stdout:192: expect (ObjNum 18 snap 0 seq_num 18) 2023-11-28T14:32:46.567 INFO:tasks.rados.rados.0.smithi055.stdout:190: finishing write tid 1 to smithi05583006-40 2023-11-28T14:32:46.567 INFO:tasks.rados.rados.0.smithi055.stdout:190: finishing write tid 2 to smithi05583006-40 2023-11-28T14:32:46.567 INFO:tasks.rados.rados.0.smithi055.stdout:190: finishing write tid 3 to smithi05583006-40 2023-11-28T14:32:46.567 INFO:tasks.rados.rados.0.smithi055.stdout:190: finishing write tid 4 to smithi05583006-40 2023-11-28T14:32:46.567 INFO:tasks.rados.rados.0.smithi055.stdout:190: finishing write tid 5 to smithi05583006-40 2023-11-28T14:32:46.567 INFO:tasks.rados.rados.0.smithi055.stdout:190: finishing write tid 6 to smithi05583006-40 2023-11-28T14:32:46.567 INFO:tasks.rados.rados.0.smithi055.stdout:190: oid 40 updating version 0 to 13 2023-11-28T14:32:46.568 INFO:tasks.rados.rados.0.smithi055.stdout:190: oid 40 version 13 is already newer than 12 2023-11-28T14:32:46.568 INFO:tasks.rados.rados.0.smithi055.stdout:190: oid 40 updating version 13 to 16 2023-11-28T14:32:46.568 INFO:tasks.rados.rados.0.smithi055.stdout:190: oid 40 version 16 is already newer than 14 2023-11-28T14:32:46.568 INFO:tasks.rados.rados.0.smithi055.stdout:190: oid 40 version 16 is already newer than 16 2023-11-28T14:32:46.568 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 40 v 16 (ObjNum 114 snap 17 seq_num 114) dirty exists 2023-11-28T14:32:46.568 INFO:tasks.rados.rados.0.smithi055.stdout:190: left oid 40 (ObjNum 114 snap 17 seq_num 114) 2023-11-28T14:32:46.950 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:32:47.064 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:47.062+0000 7f3bd6c20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:47.067 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:47.066+0000 7f3bd6c20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:47.415 INFO:tasks.rados.rados.0.smithi055.stdout:190: done (3 left) 2023-11-28T14:32:47.415 INFO:tasks.rados.rados.0.smithi055.stdout:191: done (2 left) 2023-11-28T14:32:47.416 INFO:tasks.rados.rados.0.smithi055.stdout:192: done (1 left) 2023-11-28T14:32:47.416 INFO:tasks.rados.rados.0.smithi055.stdout:193: done (0 left) 2023-11-28T14:32:47.416 INFO:tasks.rados.rados.0.smithi055.stdout:194: read oid 6 snap -1 2023-11-28T14:32:47.416 INFO:tasks.rados.rados.0.smithi055.stdout:194: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-28T14:32:47.416 INFO:tasks.rados.rados.0.smithi055.stdout:195: read oid 37 snap -1 2023-11-28T14:32:47.416 INFO:tasks.rados.rados.0.smithi055.stdout:195: expect deleted 2023-11-28T14:32:47.416 INFO:tasks.rados.rados.0.smithi055.stdout:196: write (excl) oid 17 current snap is 18 2023-11-28T14:32:47.416 INFO:tasks.rados.rados.0.smithi055.stdout:196: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2023-11-28T14:32:47.423 INFO:tasks.rados.rados.0.smithi055.stdout:196: writing smithi05583006-17 from 732805 to 1531735 tid 1 2023-11-28T14:32:47.431 INFO:tasks.rados.rados.0.smithi055.stdout:196: writing smithi05583006-17 from 2296488 to 2867514 tid 2 2023-11-28T14:32:47.437 INFO:tasks.rados.rados.0.smithi055.stdout:196: writing smithi05583006-17 from 3291997 to 3697102 tid 3 2023-11-28T14:32:47.437 INFO:tasks.rados.rados.0.smithi055.stdout:197: read oid 47 snap -1 2023-11-28T14:32:47.437 INFO:tasks.rados.rados.0.smithi055.stdout:197: expect (ObjNum 55 snap 1 seq_num 55) 2023-11-28T14:32:47.437 INFO:tasks.rados.rados.0.smithi055.stdout:198: delete oid 28 current snap is 18 2023-11-28T14:32:47.473 INFO:tasks.rados.rados.0.smithi055.stdout:194: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-28T14:32:47.561 INFO:tasks.rados.rados.0.smithi055.stdout:197: expect (ObjNum 55 snap 1 seq_num 55) 2023-11-28T14:32:47.710 INFO:tasks.rados.rados.0.smithi055.stdout:196: finishing write tid 1 to smithi05583006-17 2023-11-28T14:32:47.710 INFO:tasks.rados.rados.0.smithi055.stdout:196: finishing write tid 2 to smithi05583006-17 2023-11-28T14:32:47.710 INFO:tasks.rados.rados.0.smithi055.stdout:196: finishing write tid 3 to smithi05583006-17 2023-11-28T14:32:47.710 INFO:tasks.rados.rados.0.smithi055.stdout:196: finishing write tid 4 to smithi05583006-17 2023-11-28T14:32:47.711 INFO:tasks.rados.rados.0.smithi055.stdout:196: finishing write tid 5 to smithi05583006-17 2023-11-28T14:32:47.711 INFO:tasks.rados.rados.0.smithi055.stdout:196: oid 17 updating version 0 to 26 2023-11-28T14:32:47.711 INFO:tasks.rados.rados.0.smithi055.stdout:196: oid 17 version 26 is already newer than 24 2023-11-28T14:32:47.711 INFO:tasks.rados.rados.0.smithi055.stdout:196: oid 17 version 26 is already newer than 26 2023-11-28T14:32:47.711 INFO:tasks.rados.rados.0.smithi055.stdout:196: oid 17 version 26 is already newer than 23 2023-11-28T14:32:47.711 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 17 v 26 (ObjNum 115 snap 18 seq_num 115) dirty exists 2023-11-28T14:32:47.711 INFO:tasks.rados.rados.0.smithi055.stdout:196: left oid 17 (ObjNum 115 snap 18 seq_num 115) 2023-11-28T14:32:47.711 INFO:tasks.rados.rados.0.smithi055.stdout:194: done (4 left) 2023-11-28T14:32:47.711 INFO:tasks.rados.rados.0.smithi055.stdout:195: done (3 left) 2023-11-28T14:32:47.711 INFO:tasks.rados.rados.0.smithi055.stdout:196: done (2 left) 2023-11-28T14:32:47.711 INFO:tasks.rados.rados.0.smithi055.stdout:197: done (1 left) 2023-11-28T14:32:47.711 INFO:tasks.rados.rados.0.smithi055.stdout:198: done (0 left) 2023-11-28T14:32:47.712 INFO:tasks.rados.rados.0.smithi055.stdout:199: delete oid 43 current snap is 18 2023-11-28T14:32:47.735 INFO:tasks.rados.rados.0.smithi055.stdout:199: done (0 left) 2023-11-28T14:32:47.735 INFO:tasks.rados.rados.0.smithi055.stdout:200: write (excl) oid 24 current snap is 18 2023-11-28T14:32:47.735 INFO:tasks.rados.rados.0.smithi055.stdout:200: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2023-11-28T14:32:47.742 INFO:tasks.rados.rados.0.smithi055.stdout:200: writing smithi05583006-24 from 783226 to 1407903 tid 1 2023-11-28T14:32:47.748 INFO:tasks.rados.rados.0.smithi055.stdout:200: writing smithi05583006-24 from 2071934 to 2506640 tid 2 2023-11-28T14:32:47.755 INFO:tasks.rados.rados.0.smithi055.stdout:200: writing smithi05583006-24 from 2974266 to 3613187 tid 3 2023-11-28T14:32:47.755 INFO:tasks.rados.rados.0.smithi055.stdout:201: snap_remove snap 16 2023-11-28T14:32:48.419 INFO:tasks.rados.rados.0.smithi055.stdout:201: done (1 left) 2023-11-28T14:32:48.419 INFO:tasks.rados.rados.0.smithi055.stdout:202: write (excl) oid 49 current snap is 18 2023-11-28T14:32:48.419 INFO:tasks.rados.rados.0.smithi055.stdout:200: finishing write tid 1 to smithi05583006-24 2023-11-28T14:32:48.419 INFO:tasks.rados.rados.0.smithi055.stdout:202: seq_num 119 ranges {400033=699926,1614063=385970} 2023-11-28T14:32:48.426 INFO:tasks.rados.rados.0.smithi055.stdout:202: writing smithi05583006-49 from 400033 to 1099959 tid 1 2023-11-28T14:32:48.433 INFO:tasks.rados.rados.0.smithi055.stdout:202: writing smithi05583006-49 from 1614063 to 2000033 tid 2 2023-11-28T14:32:48.433 INFO:tasks.rados.rados.0.smithi055.stdout:203: delete oid 13 current snap is 18 2023-11-28T14:32:48.433 INFO:tasks.rados.rados.0.smithi055.stdout:200: finishing write tid 2 to smithi05583006-24 2023-11-28T14:32:48.433 INFO:tasks.rados.rados.0.smithi055.stdout:200: finishing write tid 3 to smithi05583006-24 2023-11-28T14:32:48.433 INFO:tasks.rados.rados.0.smithi055.stdout:200: finishing write tid 4 to smithi05583006-24 2023-11-28T14:32:48.433 INFO:tasks.rados.rados.0.smithi055.stdout:200: finishing write tid 5 to smithi05583006-24 2023-11-28T14:32:48.433 INFO:tasks.rados.rados.0.smithi055.stdout:200: oid 24 updating version 0 to 16 2023-11-28T14:32:48.433 INFO:tasks.rados.rados.0.smithi055.stdout:200: oid 24 updating version 16 to 18 2023-11-28T14:32:48.434 INFO:tasks.rados.rados.0.smithi055.stdout:200: oid 24 version 18 is already newer than 18 2023-11-28T14:32:48.434 INFO:tasks.rados.rados.0.smithi055.stdout:200: oid 24 version 18 is already newer than 15 2023-11-28T14:32:48.434 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 24 v 18 (ObjNum 118 snap 18 seq_num 118) dirty exists 2023-11-28T14:32:48.434 INFO:tasks.rados.rados.0.smithi055.stdout:200: left oid 24 (ObjNum 118 snap 18 seq_num 118) 2023-11-28T14:32:48.467 INFO:tasks.rados.rados.0.smithi055.stdout:200: done (2 left) 2023-11-28T14:32:48.467 INFO:tasks.rados.rados.0.smithi055.stdout:203: done (1 left) 2023-11-28T14:32:48.467 INFO:tasks.rados.rados.0.smithi055.stdout:204: delete oid 29 current snap is 18 2023-11-28T14:32:48.512 INFO:tasks.rados.rados.0.smithi055.stdout:204: done (1 left) 2023-11-28T14:32:48.512 INFO:tasks.rados.rados.0.smithi055.stdout:205: read oid 26 snap -1 2023-11-28T14:32:48.512 INFO:tasks.rados.rados.0.smithi055.stdout:205: expect deleted 2023-11-28T14:32:48.512 INFO:tasks.rados.rados.0.smithi055.stdout:206: write (excl) oid 40 current snap is 18 2023-11-28T14:32:48.513 INFO:tasks.rados.rados.0.smithi055.stdout:206: seq_num 122 ranges {450454=642026,1505862=544592} 2023-11-28T14:32:48.519 INFO:tasks.rados.rados.0.smithi055.stdout:206: writing smithi05583006-40 from 450454 to 1092480 tid 1 2023-11-28T14:32:48.524 INFO:tasks.rados.rados.0.smithi055.stdout:206: writing smithi05583006-40 from 1505862 to 2050454 tid 2 2023-11-28T14:32:48.524 INFO:tasks.rados.rados.0.smithi055.stdout:207: delete oid 4 current snap is 18 2023-11-28T14:32:48.524 INFO:tasks.rados.rados.0.smithi055.stdout:202: finishing write tid 1 to smithi05583006-49 2023-11-28T14:32:48.530 INFO:tasks.rados.rados.0.smithi055.stdout:202: finishing write tid 2 to smithi05583006-49 2023-11-28T14:32:48.559 INFO:tasks.rados.rados.0.smithi055.stdout:202: finishing write tid 3 to smithi05583006-49 2023-11-28T14:32:48.569 INFO:tasks.rados.rados.0.smithi055.stdout:202: finishing write tid 4 to smithi05583006-49 2023-11-28T14:32:48.569 INFO:tasks.rados.rados.0.smithi055.stdout:202: oid 49 updating version 0 to 32 2023-11-28T14:32:48.569 INFO:tasks.rados.rados.0.smithi055.stdout:202: oid 49 updating version 32 to 33 2023-11-28T14:32:48.569 INFO:tasks.rados.rados.0.smithi055.stdout:202: oid 49 updating version 33 to 34 2023-11-28T14:32:48.570 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 49 v 34 (ObjNum 119 snap 18 seq_num 119) dirty exists 2023-11-28T14:32:48.570 INFO:tasks.rados.rados.0.smithi055.stdout:202: left oid 49 (ObjNum 119 snap 18 seq_num 119) 2023-11-28T14:32:48.581 INFO:tasks.rados.rados.0.smithi055.stdout:202: done (3 left) 2023-11-28T14:32:48.581 INFO:tasks.rados.rados.0.smithi055.stdout:205: done (2 left) 2023-11-28T14:32:48.581 INFO:tasks.rados.rados.0.smithi055.stdout:207: done (1 left) 2023-11-28T14:32:48.581 INFO:tasks.rados.rados.0.smithi055.stdout:208: delete oid 15 current snap is 18 2023-11-28T14:32:48.616 INFO:tasks.rados.rados.0.smithi055.stdout:208: done (1 left) 2023-11-28T14:32:48.616 INFO:tasks.rados.rados.0.smithi055.stdout:209: write (excl) oid 1 current snap is 18 2023-11-28T14:32:48.616 INFO:tasks.rados.rados.0.smithi055.stdout:209: seq_num 125 ranges {500875=467773,1564955=535920} 2023-11-28T14:32:48.621 INFO:tasks.rados.rados.0.smithi055.stdout:209: writing smithi05583006-1 from 500875 to 968648 tid 1 2023-11-28T14:32:48.627 INFO:tasks.rados.rados.0.smithi055.stdout:209: writing smithi05583006-1 from 1564955 to 2100875 tid 2 2023-11-28T14:32:48.628 INFO:tasks.rados.rados.0.smithi055.stdout:210: delete oid 12 current snap is 18 2023-11-28T14:32:48.628 INFO:tasks.rados.rados.0.smithi055.stdout:206: finishing write tid 1 to smithi05583006-40 2023-11-28T14:32:48.658 INFO:tasks.rados.rados.0.smithi055.stdout:206: finishing write tid 2 to smithi05583006-40 2023-11-28T14:32:48.659 INFO:tasks.rados.rados.0.smithi055.stdout:209: finishing write tid 1 to smithi05583006-1 2023-11-28T14:32:48.690 INFO:tasks.rados.rados.0.smithi055.stdout:206: finishing write tid 3 to smithi05583006-40 2023-11-28T14:32:48.691 INFO:tasks.rados.rados.0.smithi055.stdout:210: done (2 left) 2023-11-28T14:32:48.692 INFO:tasks.rados.rados.0.smithi055.stdout:211: read oid 21 snap -1 2023-11-28T14:32:48.692 INFO:tasks.rados.rados.0.smithi055.stdout:211: expect deleted 2023-11-28T14:32:48.692 INFO:tasks.rados.rados.0.smithi055.stdout:212: read oid 6 snap -1 2023-11-28T14:32:48.692 INFO:tasks.rados.rados.0.smithi055.stdout:212: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-28T14:32:48.692 INFO:tasks.rados.rados.0.smithi055.stdout:213: write (excl) oid 22 current snap is 18 2023-11-28T14:32:48.692 INFO:tasks.rados.rados.0.smithi055.stdout:213: seq_num 127 ranges {534489=618271,1565566=568923} 2023-11-28T14:32:48.699 INFO:tasks.rados.rados.0.smithi055.stdout:213: writing smithi05583006-22 from 534489 to 1152760 tid 1 2023-11-28T14:32:48.706 INFO:tasks.rados.rados.0.smithi055.stdout:213: writing smithi05583006-22 from 1565566 to 2134489 tid 2 2023-11-28T14:32:48.706 INFO:tasks.rados.rados.0.smithi055.stdout:214: read oid 32 snap -1 2023-11-28T14:32:48.706 INFO:tasks.rados.rados.0.smithi055.stdout:214: expect (ObjNum 31 snap 0 seq_num 31) 2023-11-28T14:32:48.706 INFO:tasks.rados.rados.0.smithi055.stdout:215: write oid 15 current snap is 18 2023-11-28T14:32:48.706 INFO:tasks.rados.rados.0.smithi055.stdout:206: finishing write tid 4 to smithi05583006-40 2023-11-28T14:32:48.707 INFO:tasks.rados.rados.0.smithi055.stdout:206: oid 40 updating version 0 to 22 2023-11-28T14:32:48.707 INFO:tasks.rados.rados.0.smithi055.stdout:206: oid 40 version 22 is already newer than 20 2023-11-28T14:32:48.707 INFO:tasks.rados.rados.0.smithi055.stdout:206: oid 40 version 22 is already newer than 21 2023-11-28T14:32:48.707 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 40 v 22 (ObjNum 122 snap 18 seq_num 122) dirty exists 2023-11-28T14:32:48.707 INFO:tasks.rados.rados.0.smithi055.stdout:206: left oid 40 (ObjNum 122 snap 18 seq_num 122) 2023-11-28T14:32:48.707 INFO:tasks.rados.rados.0.smithi055.stdout:215: seq_num 128 ranges {551296=693520,2024048=127248} 2023-11-28T14:32:48.713 INFO:tasks.rados.rados.0.smithi055.stdout:215: writing smithi05583006-15 from 551296 to 1244816 tid 1 2023-11-28T14:32:48.719 INFO:tasks.rados.rados.0.smithi055.stdout:215: writing smithi05583006-15 from 2024048 to 2151296 tid 2 2023-11-28T14:32:48.719 INFO:tasks.rados.rados.0.smithi055.stdout:206: done (6 left) 2023-11-28T14:32:48.719 INFO:tasks.rados.rados.0.smithi055.stdout:216: delete oid 44 current snap is 18 2023-11-28T14:32:48.719 INFO:tasks.rados.rados.0.smithi055.stdout:209: finishing write tid 2 to smithi05583006-1 2023-11-28T14:32:48.719 INFO:tasks.rados.rados.0.smithi055.stdout:209: finishing write tid 3 to smithi05583006-1 2023-11-28T14:32:48.719 INFO:tasks.rados.rados.0.smithi055.stdout:209: finishing write tid 4 to smithi05583006-1 2023-11-28T14:32:48.719 INFO:tasks.rados.rados.0.smithi055.stdout:209: oid 1 updating version 0 to 7 2023-11-28T14:32:48.720 INFO:tasks.rados.rados.0.smithi055.stdout:209: oid 1 version 7 is already newer than 7 2023-11-28T14:32:48.720 INFO:tasks.rados.rados.0.smithi055.stdout:209: oid 1 version 7 is already newer than 5 2023-11-28T14:32:48.720 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 1 v 7 (ObjNum 125 snap 18 seq_num 125) dirty exists 2023-11-28T14:32:48.720 INFO:tasks.rados.rados.0.smithi055.stdout:209: left oid 1 (ObjNum 125 snap 18 seq_num 125) 2023-11-28T14:32:48.720 INFO:tasks.rados.rados.0.smithi055.stdout:212: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-28T14:32:48.843 INFO:tasks.rados.rados.0.smithi055.stdout:214: expect (ObjNum 31 snap 0 seq_num 31) 2023-11-28T14:32:48.944 INFO:tasks.rados.rados.0.smithi055.stdout:213: finishing write tid 1 to smithi05583006-22 2023-11-28T14:32:48.944 INFO:tasks.rados.rados.0.smithi055.stdout:215: finishing write tid 1 to smithi05583006-15 2023-11-28T14:32:48.944 INFO:tasks.rados.rados.0.smithi055.stdout:213: finishing write tid 2 to smithi05583006-22 2023-11-28T14:32:48.944 INFO:tasks.rados.rados.0.smithi055.stdout:215: finishing write tid 2 to smithi05583006-15 2023-11-28T14:32:48.945 INFO:tasks.rados.rados.0.smithi055.stdout:215: finishing write tid 3 to smithi05583006-15 2023-11-28T14:32:48.945 INFO:tasks.rados.rados.0.smithi055.stdout:215: finishing write tid 4 to smithi05583006-15 2023-11-28T14:32:48.945 INFO:tasks.rados.rados.0.smithi055.stdout:215: oid 15 updating version 0 to 23 2023-11-28T14:32:48.945 INFO:tasks.rados.rados.0.smithi055.stdout:215: oid 15 updating version 23 to 25 2023-11-28T14:32:48.945 INFO:tasks.rados.rados.0.smithi055.stdout:215: oid 15 version 25 is already newer than 25 2023-11-28T14:32:48.945 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 15 v 25 (ObjNum 128 snap 18 seq_num 128) dirty exists 2023-11-28T14:32:48.945 INFO:tasks.rados.rados.0.smithi055.stdout:215: left oid 15 (ObjNum 128 snap 18 seq_num 128) 2023-11-28T14:32:48.945 INFO:tasks.rados.rados.0.smithi055.stdout:213: finishing write tid 3 to smithi05583006-22 2023-11-28T14:32:48.945 INFO:tasks.rados.rados.0.smithi055.stdout:209: done (6 left) 2023-11-28T14:32:48.945 INFO:tasks.rados.rados.0.smithi055.stdout:211: done (5 left) 2023-11-28T14:32:48.945 INFO:tasks.rados.rados.0.smithi055.stdout:212: done (4 left) 2023-11-28T14:32:48.945 INFO:tasks.rados.rados.0.smithi055.stdout:214: done (3 left) 2023-11-28T14:32:48.946 INFO:tasks.rados.rados.0.smithi055.stdout:215: done (2 left) 2023-11-28T14:32:48.946 INFO:tasks.rados.rados.0.smithi055.stdout:216: done (1 left) 2023-11-28T14:32:48.946 INFO:tasks.rados.rados.0.smithi055.stdout:217: read oid 35 snap -1 2023-11-28T14:32:48.946 INFO:tasks.rados.rados.0.smithi055.stdout:217: expect (ObjNum 86 snap 4 seq_num 86) 2023-11-28T14:32:48.946 INFO:tasks.rados.rados.0.smithi055.stdout:218: write oid 39 current snap is 18 2023-11-28T14:32:48.946 INFO:tasks.rados.rados.0.smithi055.stdout:213: finishing write tid 4 to smithi05583006-22 2023-11-28T14:32:48.946 INFO:tasks.rados.rados.0.smithi055.stdout:213: oid 22 updating version 0 to 109 2023-11-28T14:32:48.946 INFO:tasks.rados.rados.0.smithi055.stdout:213: oid 22 version 109 is already newer than 108 2023-11-28T14:32:48.946 INFO:tasks.rados.rados.0.smithi055.stdout:213: oid 22 updating version 109 to 110 2023-11-28T14:32:48.946 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 22 v 110 (ObjNum 127 snap 18 seq_num 127) dirty exists 2023-11-28T14:32:48.946 INFO:tasks.rados.rados.0.smithi055.stdout:213: left oid 22 (ObjNum 127 snap 18 seq_num 127) 2023-11-28T14:32:48.947 INFO:tasks.rados.rados.0.smithi055.stdout:218: seq_num 130 ranges {584910=560371,1857365=327545} 2023-11-28T14:32:48.949 INFO:tasks.rados.rados.0.smithi055.stdout:218: writing smithi05583006-39 from 584910 to 1145281 tid 1 2023-11-28T14:32:48.956 INFO:tasks.rados.rados.0.smithi055.stdout:218: writing smithi05583006-39 from 1857365 to 2184910 tid 2 2023-11-28T14:32:48.956 INFO:tasks.rados.rados.0.smithi055.stdout:213: done (2 left) 2023-11-28T14:32:48.957 INFO:tasks.rados.rados.0.smithi055.stdout:219: delete oid 11 current snap is 18 2023-11-28T14:32:48.965 INFO:tasks.rados.rados.0.smithi055.stdout:217: expect (ObjNum 86 snap 4 seq_num 86) 2023-11-28T14:32:49.139 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:32:49.189 INFO:tasks.rados.rados.0.smithi055.stdout:218: finishing write tid 1 to smithi05583006-39 2023-11-28T14:32:49.190 INFO:tasks.rados.rados.0.smithi055.stdout:218: finishing write tid 2 to smithi05583006-39 2023-11-28T14:32:49.190 INFO:tasks.rados.rados.0.smithi055.stdout:218: finishing write tid 3 to smithi05583006-39 2023-11-28T14:32:49.190 INFO:tasks.rados.rados.0.smithi055.stdout:218: finishing write tid 4 to smithi05583006-39 2023-11-28T14:32:49.190 INFO:tasks.rados.rados.0.smithi055.stdout:218: oid 39 updating version 0 to 22 2023-11-28T14:32:49.190 INFO:tasks.rados.rados.0.smithi055.stdout:218: oid 39 version 22 is already newer than 21 2023-11-28T14:32:49.190 INFO:tasks.rados.rados.0.smithi055.stdout:218: oid 39 updating version 22 to 23 2023-11-28T14:32:49.190 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 39 v 23 (ObjNum 130 snap 18 seq_num 130) dirty exists 2023-11-28T14:32:49.190 INFO:tasks.rados.rados.0.smithi055.stdout:218: left oid 39 (ObjNum 130 snap 18 seq_num 130) 2023-11-28T14:32:49.190 INFO:tasks.rados.rados.0.smithi055.stdout:217: done (2 left) 2023-11-28T14:32:49.190 INFO:tasks.rados.rados.0.smithi055.stdout:218: done (1 left) 2023-11-28T14:32:49.190 INFO:tasks.rados.rados.0.smithi055.stdout:219: done (0 left) 2023-11-28T14:32:49.190 INFO:tasks.rados.rados.0.smithi055.stdout:220: snap_remove snap 15 2023-11-28T14:32:49.248 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:49.246+0000 7f16a779e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:49.251 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:49.250+0000 7f16a779e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:49.412 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:32:49.412 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:32:49.412 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-11-28T14:32:49.412 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-11-28T14:32:49.425 INFO:tasks.rados.rados.0.smithi055.stdout:220: done (0 left) 2023-11-28T14:32:49.425 INFO:tasks.rados.rados.0.smithi055.stdout:221: read oid 11 snap -1 2023-11-28T14:32:49.425 INFO:tasks.rados.rados.0.smithi055.stdout:221: expect deleted 2023-11-28T14:32:49.425 INFO:tasks.rados.rados.0.smithi055.stdout:222: read oid 33 snap -1 2023-11-28T14:32:49.425 INFO:tasks.rados.rados.0.smithi055.stdout:222: expect (ObjNum 103 snap 11 seq_num 103) 2023-11-28T14:32:49.425 INFO:tasks.rados.rados.0.smithi055.stdout:223: write oid 22 current snap is 18 2023-11-28T14:32:49.425 INFO:tasks.rados.rados.0.smithi055.stdout:223: seq_num 132 ranges {618524=710869,1857976=360548} 2023-11-28T14:32:49.434 INFO:tasks.rados.rados.0.smithi055.stdout:223: writing smithi05583006-22 from 618524 to 1329393 tid 1 2023-11-28T14:32:49.440 INFO:tasks.rados.rados.0.smithi055.stdout:223: writing smithi05583006-22 from 1857976 to 2218524 tid 2 2023-11-28T14:32:49.441 INFO:tasks.rados.rados.0.smithi055.stdout:224: delete oid 12 current snap is 18 2023-11-28T14:32:49.491 INFO:tasks.rados.rados.0.smithi055.stdout:221: done (3 left) 2023-11-28T14:32:49.491 INFO:tasks.rados.rados.0.smithi055.stdout:224: done (2 left) 2023-11-28T14:32:49.491 INFO:tasks.rados.rados.0.smithi055.stdout:225: read oid 31 snap -1 2023-11-28T14:32:49.491 INFO:tasks.rados.rados.0.smithi055.stdout:225: expect (ObjNum 73 snap 2 seq_num 73) 2023-11-28T14:32:49.491 INFO:tasks.rados.rados.0.smithi055.stdout:226: write oid 24 current snap is 18 2023-11-28T14:32:49.492 INFO:tasks.rados.rados.0.smithi055.stdout:226: seq_num 134 ranges {652138=461367,1574940=677198} 2023-11-28T14:32:49.497 INFO:tasks.rados.rados.0.smithi055.stdout:226: writing smithi05583006-24 from 652138 to 1113505 tid 1 2023-11-28T14:32:49.503 INFO:tasks.rados.rados.0.smithi055.stdout:226: writing smithi05583006-24 from 1574940 to 2252138 tid 2 2023-11-28T14:32:49.503 INFO:tasks.rados.rados.0.smithi055.stdout:227: snap_remove snap 17 2023-11-28T14:32:49.552 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:49.546+0000 7fc3c526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:49.556 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:49.555+0000 7fc3c526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:49.595 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:49.596 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:49.596 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:49.596 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:49.596 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:49.596 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:49.596 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:49.596 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:49.596 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:49.596 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:49.596 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:49.596 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:49.599 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:49.599 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:49.600 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-11-28T14:32:49.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:49.748+0000 7fcba066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:49.756 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:49.755+0000 7fcba066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:49.768 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:49.769 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:49.769 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:49.769 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:49.769 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:49.769 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:49.769 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:49.769 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:49.769 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:49.769 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:49.769 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:49.769 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:49.772 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:49.772 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:49.773 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-11-28T14:32:49.885 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:49.880+0000 7f144c99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:49.888 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:49.887+0000 7f144c99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:49.901 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:49.901 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:49.901 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:49.901 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:49.901 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:49.901 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:49.902 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:49.902 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:49.902 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:49.902 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:49.902 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:49.902 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:49.904 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:49.905 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-11-28T14:32:50.019 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:50.013+0000 7f61db11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:50.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:50.021+0000 7f61db11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:50.034 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:32:50.034 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:32:50.034 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:32:50.034 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:32:50.035 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:32:50.035 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:32:50.035 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:32:50.035 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:32:50.035 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:32:50.035 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:32:50.035 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:32:50.035 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:32:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:32:50.038 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:32:50.428 INFO:tasks.rados.rados.0.smithi055.stdout:227: done (4 left) 2023-11-28T14:32:50.428 INFO:tasks.rados.rados.0.smithi055.stdout:228: read oid 43 snap -1 2023-11-28T14:32:50.428 INFO:tasks.rados.rados.0.smithi055.stdout:228: expect deleted 2023-11-28T14:32:50.428 INFO:tasks.rados.rados.0.smithi055.stdout:229: read oid 26 snap -1 2023-11-28T14:32:50.428 INFO:tasks.rados.rados.0.smithi055.stdout:229: expect deleted 2023-11-28T14:32:50.428 INFO:tasks.rados.rados.0.smithi055.stdout:230: read oid 11 snap -1 2023-11-28T14:32:50.428 INFO:tasks.rados.rados.0.smithi055.stdout:230: expect deleted 2023-11-28T14:32:50.428 INFO:tasks.rados.rados.0.smithi055.stdout:231: write (excl) oid 44 current snap is 18 2023-11-28T14:32:50.428 INFO:tasks.rados.rados.0.smithi055.stdout:231: seq_num 135 ranges {668945=536616,1633422=491363} 2023-11-28T14:32:50.433 INFO:tasks.rados.rados.0.smithi055.stdout:231: writing smithi05583006-44 from 668945 to 1205561 tid 1 2023-11-28T14:32:50.439 INFO:tasks.rados.rados.0.smithi055.stdout:231: writing smithi05583006-44 from 1633422 to 2124785 tid 2 2023-11-28T14:32:50.439 INFO:tasks.rados.rados.0.smithi055.stdout:232: snap_create 2023-11-28T14:32:50.439 INFO:tasks.rados.rados.0.smithi055.stdout:222: expect (ObjNum 103 snap 11 seq_num 103) 2023-11-28T14:32:50.730 INFO:tasks.rados.rados.0.smithi055.stdout:225: expect (ObjNum 73 snap 2 seq_num 73) 2023-11-28T14:32:50.955 INFO:tasks.rados.rados.0.smithi055.stdout:226: finishing write tid 1 to smithi05583006-24 2023-11-28T14:32:50.955 INFO:tasks.rados.rados.0.smithi055.stdout:223: finishing write tid 1 to smithi05583006-22 2023-11-28T14:32:50.955 INFO:tasks.rados.rados.0.smithi055.stdout:226: finishing write tid 2 to smithi05583006-24 2023-11-28T14:32:50.955 INFO:tasks.rados.rados.0.smithi055.stdout:223: finishing write tid 2 to smithi05583006-22 2023-11-28T14:32:50.955 INFO:tasks.rados.rados.0.smithi055.stdout:223: finishing write tid 3 to smithi05583006-22 2023-11-28T14:32:50.955 INFO:tasks.rados.rados.0.smithi055.stdout:226: finishing write tid 3 to smithi05583006-24 2023-11-28T14:32:50.956 INFO:tasks.rados.rados.0.smithi055.stdout:226: finishing write tid 4 to smithi05583006-24 2023-11-28T14:32:50.956 INFO:tasks.rados.rados.0.smithi055.stdout:226: oid 24 updating version 0 to 25 2023-11-28T14:32:50.956 INFO:tasks.rados.rados.0.smithi055.stdout:226: oid 24 updating version 25 to 26 2023-11-28T14:32:50.956 INFO:tasks.rados.rados.0.smithi055.stdout:226: oid 24 updating version 26 to 27 2023-11-28T14:32:50.956 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 24 v 27 (ObjNum 134 snap 18 seq_num 134) dirty exists 2023-11-28T14:32:50.956 INFO:tasks.rados.rados.0.smithi055.stdout:226: left oid 24 (ObjNum 134 snap 18 seq_num 134) 2023-11-28T14:32:50.956 INFO:tasks.rados.rados.0.smithi055.stdout:223: finishing write tid 4 to smithi05583006-22 2023-11-28T14:32:50.956 INFO:tasks.rados.rados.0.smithi055.stdout:223: oid 22 updating version 0 to 117 2023-11-28T14:32:50.956 INFO:tasks.rados.rados.0.smithi055.stdout:223: oid 22 updating version 117 to 119 2023-11-28T14:32:50.956 INFO:tasks.rados.rados.0.smithi055.stdout:223: oid 22 version 119 is already newer than 119 2023-11-28T14:32:50.956 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 22 v 119 (ObjNum 132 snap 18 seq_num 132) dirty exists 2023-11-28T14:32:50.956 INFO:tasks.rados.rados.0.smithi055.stdout:223: left oid 22 (ObjNum 132 snap 18 seq_num 132) 2023-11-28T14:32:50.957 INFO:tasks.rados.rados.0.smithi055.stdout:231: finishing write tid 1 to smithi05583006-44 2023-11-28T14:32:50.957 INFO:tasks.rados.rados.0.smithi055.stdout:231: finishing write tid 2 to smithi05583006-44 2023-11-28T14:32:50.957 INFO:tasks.rados.rados.0.smithi055.stdout:231: finishing write tid 3 to smithi05583006-44 2023-11-28T14:32:50.957 INFO:tasks.rados.rados.0.smithi055.stdout:231: finishing write tid 4 to smithi05583006-44 2023-11-28T14:32:50.957 INFO:tasks.rados.rados.0.smithi055.stdout:231: oid 44 updating version 0 to 125 2023-11-28T14:32:50.957 INFO:tasks.rados.rados.0.smithi055.stdout:231: oid 44 version 125 is already newer than 124 2023-11-28T14:32:50.957 INFO:tasks.rados.rados.0.smithi055.stdout:231: oid 44 updating version 125 to 126 2023-11-28T14:32:50.957 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 44 v 126 (ObjNum 135 snap 18 seq_num 135) dirty exists 2023-11-28T14:32:50.957 INFO:tasks.rados.rados.0.smithi055.stdout:231: left oid 44 (ObjNum 135 snap 18 seq_num 135) 2023-11-28T14:32:51.321 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:32:51.435 INFO:tasks.rados.rados.0.smithi055.stdout:222: done (8 left) 2023-11-28T14:32:51.436 INFO:tasks.rados.rados.0.smithi055.stdout:223: done (7 left) 2023-11-28T14:32:51.436 INFO:tasks.rados.rados.0.smithi055.stdout:225: done (6 left) 2023-11-28T14:32:51.436 INFO:tasks.rados.rados.0.smithi055.stdout:226: done (5 left) 2023-11-28T14:32:51.436 INFO:tasks.rados.rados.0.smithi055.stdout:228: done (4 left) 2023-11-28T14:32:51.436 INFO:tasks.rados.rados.0.smithi055.stdout:229: done (3 left) 2023-11-28T14:32:51.436 INFO:tasks.rados.rados.0.smithi055.stdout:230: done (2 left) 2023-11-28T14:32:51.436 INFO:tasks.rados.rados.0.smithi055.stdout:231: done (1 left) 2023-11-28T14:32:51.436 INFO:tasks.rados.rados.0.smithi055.stdout:232: done (0 left) 2023-11-28T14:32:51.436 INFO:tasks.rados.rados.0.smithi055.stdout:233: read oid 49 snap -1 2023-11-28T14:32:51.436 INFO:tasks.rados.rados.0.smithi055.stdout:233: expect (ObjNum 119 snap 18 seq_num 119) 2023-11-28T14:32:51.436 INFO:tasks.rados.rados.0.smithi055.stdout:234: write (excl) oid 31 current snap is 19 2023-11-28T14:32:51.437 INFO:tasks.rados.rados.0.smithi055.stdout:234: seq_num 136 ranges {685752=611865,1975551=310201} 2023-11-28T14:32:51.438 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:51.436+0000 7f13d0715640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:51.442 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:51.441+0000 7f13d0715640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:51.444 INFO:tasks.rados.rados.0.smithi055.stdout:234: writing smithi05583006-31 from 685752 to 1297617 tid 1 2023-11-28T14:32:51.450 INFO:tasks.rados.rados.0.smithi055.stdout:234: writing smithi05583006-31 from 1975551 to 2285752 tid 2 2023-11-28T14:32:51.451 INFO:tasks.rados.rados.0.smithi055.stdout:235: snap_create 2023-11-28T14:32:51.492 INFO:tasks.rados.rados.0.smithi055.stdout:233: expect (ObjNum 119 snap 18 seq_num 119) 2023-11-28T14:32:51.578 INFO:tasks.rados.rados.0.smithi055.stdout:234: finishing write tid 1 to smithi05583006-31 2023-11-28T14:32:51.578 INFO:tasks.rados.rados.0.smithi055.stdout:234: finishing write tid 2 to smithi05583006-31 2023-11-28T14:32:51.578 INFO:tasks.rados.rados.0.smithi055.stdout:234: finishing write tid 3 to smithi05583006-31 2023-11-28T14:32:51.578 INFO:tasks.rados.rados.0.smithi055.stdout:234: finishing write tid 4 to smithi05583006-31 2023-11-28T14:32:51.578 INFO:tasks.rados.rados.0.smithi055.stdout:234: oid 31 updating version 0 to 29 2023-11-28T14:32:51.578 INFO:tasks.rados.rados.0.smithi055.stdout:234: oid 31 updating version 29 to 30 2023-11-28T14:32:51.578 INFO:tasks.rados.rados.0.smithi055.stdout:234: oid 31 updating version 30 to 31 2023-11-28T14:32:51.579 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 31 v 31 (ObjNum 136 snap 19 seq_num 136) dirty exists 2023-11-28T14:32:51.579 INFO:tasks.rados.rados.0.smithi055.stdout:234: left oid 31 (ObjNum 136 snap 19 seq_num 136) 2023-11-28T14:32:52.438 INFO:tasks.rados.rados.0.smithi055.stdout:233: done (2 left) 2023-11-28T14:32:52.439 INFO:tasks.rados.rados.0.smithi055.stdout:234: done (1 left) 2023-11-28T14:32:52.439 INFO:tasks.rados.rados.0.smithi055.stdout:235: done (0 left) 2023-11-28T14:32:52.439 INFO:tasks.rados.rados.0.smithi055.stdout:236: snap_create 2023-11-28T14:32:53.444 INFO:tasks.rados.rados.0.smithi055.stdout:236: done (0 left) 2023-11-28T14:32:53.444 INFO:tasks.rados.rados.0.smithi055.stdout:237: read oid 44 snap -1 2023-11-28T14:32:53.444 INFO:tasks.rados.rados.0.smithi055.stdout:237: expect (ObjNum 135 snap 18 seq_num 135) 2023-11-28T14:32:53.444 INFO:tasks.rados.rados.0.smithi055.stdout:238: read oid 37 snap -1 2023-11-28T14:32:53.445 INFO:tasks.rados.rados.0.smithi055.stdout:238: expect deleted 2023-11-28T14:32:53.445 INFO:tasks.rados.rados.0.smithi055.stdout:239: write (excl) oid 7 current snap is 21 2023-11-28T14:32:53.445 INFO:tasks.rados.rados.0.smithi055.stdout:239: seq_num 137 ranges {702559=403467,1750386=495032} 2023-11-28T14:32:53.450 INFO:tasks.rados.rados.0.smithi055.stdout:239: writing smithi05583006-7 from 702559 to 1106026 tid 1 2023-11-28T14:32:53.457 INFO:tasks.rados.rados.0.smithi055.stdout:239: writing smithi05583006-7 from 1750386 to 2245418 tid 2 2023-11-28T14:32:53.457 INFO:tasks.rados.rados.0.smithi055.stdout:240: delete oid 2 current snap is 21 2023-11-28T14:32:53.534 INFO:tasks.rados.rados.0.smithi055.stdout:237: expect (ObjNum 135 snap 18 seq_num 135) 2023-11-28T14:32:53.554 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:32:53.622 INFO:tasks.rados.rados.0.smithi055.stdout:239: finishing write tid 1 to smithi05583006-7 2023-11-28T14:32:53.623 INFO:tasks.rados.rados.0.smithi055.stdout:239: finishing write tid 2 to smithi05583006-7 2023-11-28T14:32:53.623 INFO:tasks.rados.rados.0.smithi055.stdout:239: finishing write tid 3 to smithi05583006-7 2023-11-28T14:32:53.623 INFO:tasks.rados.rados.0.smithi055.stdout:239: finishing write tid 4 to smithi05583006-7 2023-11-28T14:32:53.623 INFO:tasks.rados.rados.0.smithi055.stdout:239: oid 7 updating version 0 to 26 2023-11-28T14:32:53.623 INFO:tasks.rados.rados.0.smithi055.stdout:239: oid 7 updating version 26 to 27 2023-11-28T14:32:53.623 INFO:tasks.rados.rados.0.smithi055.stdout:239: oid 7 version 27 is already newer than 25 2023-11-28T14:32:53.623 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 7 v 27 (ObjNum 137 snap 21 seq_num 137) dirty exists 2023-11-28T14:32:53.624 INFO:tasks.rados.rados.0.smithi055.stdout:239: left oid 7 (ObjNum 137 snap 21 seq_num 137) 2023-11-28T14:32:53.624 INFO:tasks.rados.rados.0.smithi055.stdout:237: done (3 left) 2023-11-28T14:32:53.624 INFO:tasks.rados.rados.0.smithi055.stdout:238: done (2 left) 2023-11-28T14:32:53.624 INFO:tasks.rados.rados.0.smithi055.stdout:239: done (1 left) 2023-11-28T14:32:53.624 INFO:tasks.rados.rados.0.smithi055.stdout:240: done (0 left) 2023-11-28T14:32:53.625 INFO:tasks.rados.rados.0.smithi055.stdout:241: snap_remove snap 18 2023-11-28T14:32:53.667 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:53.665+0000 7f02cf02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:53.670 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:53.669+0000 7f02cf02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:54.446 INFO:tasks.rados.rados.0.smithi055.stdout:241: done (0 left) 2023-11-28T14:32:54.446 INFO:tasks.rados.rados.0.smithi055.stdout:242: delete oid 14 current snap is 21 2023-11-28T14:32:54.502 INFO:tasks.rados.rados.0.smithi055.stdout:242: done (0 left) 2023-11-28T14:32:54.502 INFO:tasks.rados.rados.0.smithi055.stdout:243: snap_remove snap 19 2023-11-28T14:32:55.039 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:32:55.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:32:55.039 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:32:55.167 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:55.162+0000 7f822aead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:55.170 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:55.169+0000 7f822aead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:55.353 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:32:55.353 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:32:55.368 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2023-11-28T14:32:54.468174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":89565867,"num_objects":102,"num_object_clones":55,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":468,"num_read_kb":215353,"num_write":610,"num_write_kb":137475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":264172,"kb_used_data":156876,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377223188,"statfs":{"total":386547056640,"available":386276544512,"internally_reserved":0,"allocated":160641024,"data_stored":163237297,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17277233,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":43,"num_read_kb":34138,"num_write":37,"num_write_kb":9433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000863"},"pg_stats":[{"pgid":"3.f","version":"55'126","reported_seq":205,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-28T14:32:51.454034+0000","last_change":"2023-11-28T14:32:50.680035+0000","last_active":"2023-11-28T14:32:51.454034+0000","last_peered":"2023-11-28T14:32:51.454034+0000","last_clean":"2023-11-28T14:32:51.454034+0000","last_became_active":"2023-11-28T14:32:09.321992+0000","last_became_peered":"2023-11-28T14:32:09.321992+0000","last_unstale":"2023-11-28T14:32:51.454034+0000","last_undegraded":"2023-11-28T14:32:51.454034+0000","last_fullsized":"2023-11-28T14:32:51.454034+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15479148,"num_objects":18,"num_object_clones":11,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":63,"num_read_kb":34851,"num_write":98,"num_write_kb":28657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'26","reported_seq":117,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.478264+0000","last_change":"2023-11-28T14:32:50.572663+0000","last_active":"2023-11-28T14:32:53.478264+0000","last_peered":"2023-11-28T14:32:53.478264+0000","last_clean":"2023-11-28T14:32:53.478264+0000","last_became_active":"2023-11-28T14:32:09.308848+0000","last_became_peered":"2023-11-28T14:32:09.308848+0000","last_unstale":"2023-11-28T14:32:53.478264+0000","last_undegraded":"2023-11-28T14:32:53.478264+0000","last_fullsized":"2023-11-28T14:32:53.478264+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050454,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":6436,"num_write":19,"num_write_kb":5379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"55'31","reported_seq":125,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.497719+0000","last_change":"2023-11-28T14:32:50.632753+0000","last_active":"2023-11-28T14:32:53.497719+0000","last_peered":"2023-11-28T14:32:53.497719+0000","last_clean":"2023-11-28T14:32:53.497719+0000","last_became_active":"2023-11-28T14:32:09.337560+0000","last_became_peered":"2023-11-28T14:32:09.337560+0000","last_unstale":"2023-11-28T14:32:53.497719+0000","last_undegraded":"2023-11-28T14:32:53.497719+0000","last_fullsized":"2023-11-28T14:32:53.497719+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4319366,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":24,"num_read_kb":24014,"num_write":20,"num_write_kb":5202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"55'40","reported_seq":127,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-28T14:32:51.447403+0000","last_change":"2023-11-28T14:32:50.629744+0000","last_active":"2023-11-28T14:32:51.447403+0000","last_peered":"2023-11-28T14:32:51.447403+0000","last_clean":"2023-11-28T14:32:51.447403+0000","last_became_active":"2023-11-28T14:32:09.344782+0000","last_became_peered":"2023-11-28T14:32:09.344782+0000","last_unstale":"2023-11-28T14:32:51.447403+0000","last_undegraded":"2023-11-28T14:32:51.447403+0000","last_fullsized":"2023-11-28T14:32:51.447403+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764735,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":28,"num_read_kb":16797,"num_write":26,"num_write_kb":7877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'23","reported_seq":130,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.479210+0000","last_change":"2023-11-28T14:32:50.503231+0000","last_active":"2023-11-28T14:32:53.479210+0000","last_peered":"2023-11-28T14:32:53.479210+0000","last_clean":"2023-11-28T14:32:53.479210+0000","last_became_active":"2023-11-28T14:32:09.326269+0000","last_became_peered":"2023-11-28T14:32:09.326269+0000","last_unstale":"2023-11-28T14:32:53.479210+0000","last_undegraded":"2023-11-28T14:32:53.479210+0000","last_fullsized":"2023-11-28T14:32:53.479210+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8470629,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":22056,"num_write":27,"num_write_kb":7574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"55'33","reported_seq":116,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-28T14:32:51.445237+0000","last_change":"2023-11-28T14:32:50.656650+0000","last_active":"2023-11-28T14:32:51.445237+0000","last_peered":"2023-11-28T14:32:51.445237+0000","last_clean":"2023-11-28T14:32:51.445237+0000","last_became_active":"2023-11-28T14:32:09.308846+0000","last_became_peered":"2023-11-28T14:32:09.308846+0000","last_unstale":"2023-11-28T14:32:51.445237+0000","last_undegraded":"2023-11-28T14:32:51.445237+0000","last_fullsized":"2023-11-28T14:32:51.445237+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4437015,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":15,"num_read_kb":5568,"num_write":21,"num_write_kb":5426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"56'31","reported_seq":123,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-28T14:32:51.510011+0000","last_change":"2023-11-28T14:32:50.459730+0000","last_active":"2023-11-28T14:32:51.510011+0000","last_peered":"2023-11-28T14:32:51.510011+0000","last_clean":"2023-11-28T14:32:51.510011+0000","last_became_active":"2023-11-28T14:32:09.316660+0000","last_became_peered":"2023-11-28T14:32:09.316660+0000","last_unstale":"2023-11-28T14:32:51.510011+0000","last_undegraded":"2023-11-28T14:32:51.510011+0000","last_fullsized":"2023-11-28T14:32:51.510011+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2285752,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":15,"num_read_kb":13968,"num_write":27,"num_write_kb":7610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"45'29","reported_seq":122,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.504770+0000","last_change":"2023-11-28T14:32:50.469195+0000","last_active":"2023-11-28T14:32:53.504770+0000","last_peered":"2023-11-28T14:32:53.504770+0000","last_clean":"2023-11-28T14:32:53.504770+0000","last_became_active":"2023-11-28T14:32:09.358025+0000","last_became_peered":"2023-11-28T14:32:09.358025+0000","last_unstale":"2023-11-28T14:32:53.504770+0000","last_undegraded":"2023-11-28T14:32:53.504770+0000","last_fullsized":"2023-11-28T14:32:53.504770+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5680700,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":19,"num_read_kb":10485,"num_write":21,"num_write_kb":5738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":216,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.494555+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:32:53.494555+0000","last_peered":"2023-11-28T14:32:53.494555+0000","last_clean":"2023-11-28T14:32:53.494555+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:32:53.494555+0000","last_undegraded":"2023-11-28T14:32:53.494555+0000","last_fullsized":"2023-11-28T14:32:53.494555+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":77,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.498596+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:32:53.498596+0000","last_peered":"2023-11-28T14:32:53.498596+0000","last_clean":"2023-11-28T14:32:53.498596+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:32:53.498596+0000","last_undegraded":"2023-11-28T14:32:53.498596+0000","last_fullsized":"2023-11-28T14:32:53.498596+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"54'97","reported_seq":194,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.492513+0000","last_change":"2023-11-28T14:32:50.457991+0000","last_active":"2023-11-28T14:32:53.492513+0000","last_peered":"2023-11-28T14:32:53.492513+0000","last_clean":"2023-11-28T14:32:53.492513+0000","last_became_active":"2023-11-28T14:32:09.335805+0000","last_became_peered":"2023-11-28T14:32:09.335805+0000","last_unstale":"2023-11-28T14:32:53.492513+0000","last_undegraded":"2023-11-28T14:32:53.492513+0000","last_fullsized":"2023-11-28T14:32:53.492513+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15126135,"num_objects":15,"num_object_clones":8,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":70,"num_read_kb":37491,"num_write":76,"num_write_kb":21255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":75,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-28T14:32:51.454915+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:32:51.454915+0000","last_peered":"2023-11-28T14:32:51.454915+0000","last_clean":"2023-11-28T14:32:51.454915+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:32:51.454915+0000","last_undegraded":"2023-11-28T14:32:51.454915+0000","last_fullsized":"2023-11-28T14:32:51.454915+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"45'39","reported_seq":129,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.483372+0000","last_change":"2023-11-28T14:32:50.479619+0000","last_active":"2023-11-28T14:32:53.483372+0000","last_peered":"2023-11-28T14:32:53.483372+0000","last_clean":"2023-11-28T14:32:53.483372+0000","last_became_active":"2023-11-28T14:32:09.350412+0000","last_became_peered":"2023-11-28T14:32:09.350412+0000","last_unstale":"2023-11-28T14:32:53.483372+0000","last_undegraded":"2023-11-28T14:32:53.483372+0000","last_fullsized":"2023-11-28T14:32:53.483372+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6000033,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":20,"num_read_kb":8138,"num_write":26,"num_write_kb":7982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":77,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.493443+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:32:53.493443+0000","last_peered":"2023-11-28T14:32:53.493443+0000","last_clean":"2023-11-28T14:32:53.493443+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:32:53.493443+0000","last_undegraded":"2023-11-28T14:32:53.493443+0000","last_fullsized":"2023-11-28T14:32:53.493443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'32","reported_seq":114,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.482195+0000","last_change":"2023-11-28T14:32:50.577768+0000","last_active":"2023-11-28T14:32:53.482195+0000","last_peered":"2023-11-28T14:32:53.482195+0000","last_clean":"2023-11-28T14:32:53.482195+0000","last_became_active":"2023-11-28T14:32:09.343664+0000","last_became_peered":"2023-11-28T14:32:09.343664+0000","last_unstale":"2023-11-28T14:32:53.482195+0000","last_undegraded":"2023-11-28T14:32:53.482195+0000","last_fullsized":"2023-11-28T14:32:53.482195+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932805,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":19,"num_write_kb":5527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":75,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-28T14:32:51.443958+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:32:51.443958+0000","last_peered":"2023-11-28T14:32:51.443958+0000","last_clean":"2023-11-28T14:32:51.443958+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:32:51.443958+0000","last_undegraded":"2023-11-28T14:32:51.443958+0000","last_fullsized":"2023-11-28T14:32:51.443958+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'11","reported_seq":106,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.513488+0000","last_change":"2023-11-28T14:32:50.628771+0000","last_active":"2023-11-28T14:32:53.513488+0000","last_peered":"2023-11-28T14:32:53.513488+0000","last_clean":"2023-11-28T14:32:53.513488+0000","last_became_active":"2023-11-28T14:32:09.311653+0000","last_became_peered":"2023-11-28T14:32:09.311653+0000","last_unstale":"2023-11-28T14:32:53.513488+0000","last_undegraded":"2023-11-28T14:32:53.513488+0000","last_fullsized":"2023-11-28T14:32:53.513488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2100875,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5912,"num_write":8,"num_write_kb":1910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"55'32","reported_seq":114,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.480892+0000","last_change":"2023-11-28T14:32:50.611174+0000","last_active":"2023-11-28T14:32:53.480892+0000","last_peered":"2023-11-28T14:32:53.480892+0000","last_clean":"2023-11-28T14:32:53.480892+0000","last_became_active":"2023-11-28T14:32:09.316944+0000","last_became_peered":"2023-11-28T14:32:09.316944+0000","last_unstale":"2023-11-28T14:32:53.480892+0000","last_undegraded":"2023-11-28T14:32:53.480892+0000","last_fullsized":"2023-11-28T14:32:53.480892+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":9,"num_read_kb":5,"num_write":21,"num_write_kb":6458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":83,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.487508+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:32:53.487508+0000","last_peered":"2023-11-28T14:32:53.487508+0000","last_clean":"2023-11-28T14:32:53.487508+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:32:53.487508+0000","last_undegraded":"2023-11-28T14:32:53.487508+0000","last_fullsized":"2023-11-28T14:32:53.487508+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"55'29","reported_seq":121,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.484786+0000","last_change":"2023-11-28T14:32:50.581660+0000","last_active":"2023-11-28T14:32:53.484786+0000","last_peered":"2023-11-28T14:32:53.484786+0000","last_clean":"2023-11-28T14:32:53.484786+0000","last_became_active":"2023-11-28T14:32:09.309510+0000","last_became_peered":"2023-11-28T14:32:09.309510+0000","last_unstale":"2023-11-28T14:32:53.484786+0000","last_undegraded":"2023-11-28T14:32:53.484786+0000","last_fullsized":"2023-11-28T14:32:53.484786+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252138,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":7924,"num_write":21,"num_write_kb":6086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":83,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.479632+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:32:53.479632+0000","last_peered":"2023-11-28T14:32:53.479632+0000","last_clean":"2023-11-28T14:32:53.479632+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:32:53.479632+0000","last_undegraded":"2023-11-28T14:32:53.479632+0000","last_fullsized":"2023-11-28T14:32:53.479632+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"53'34","reported_seq":133,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.478996+0000","last_change":"2023-11-28T14:32:50.502327+0000","last_active":"2023-11-28T14:32:53.478996+0000","last_peered":"2023-11-28T14:32:53.478996+0000","last_clean":"2023-11-28T14:32:53.478996+0000","last_became_active":"2023-11-28T14:32:09.356991+0000","last_became_peered":"2023-11-28T14:32:09.356991+0000","last_unstale":"2023-11-28T14:32:53.478996+0000","last_undegraded":"2023-11-28T14:32:53.478996+0000","last_fullsized":"2023-11-28T14:32:53.478996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4823576,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":19,"num_read_kb":11725,"num_write":30,"num_write_kb":9213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":83,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.506992+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:32:53.506992+0000","last_peered":"2023-11-28T14:32:53.506992+0000","last_clean":"2023-11-28T14:32:53.506992+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:32:53.506992+0000","last_undegraded":"2023-11-28T14:32:53.506992+0000","last_fullsized":"2023-11-28T14:32:53.506992+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"45'15","reported_seq":114,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.503689+0000","last_change":"2023-11-28T14:32:50.473737+0000","last_active":"2023-11-28T14:32:53.503689+0000","last_peered":"2023-11-28T14:32:53.503689+0000","last_clean":"2023-11-28T14:32:53.503689+0000","last_became_active":"2023-11-28T14:32:09.337548+0000","last_became_peered":"2023-11-28T14:32:09.337548+0000","last_unstale":"2023-11-28T14:32:53.503689+0000","last_undegraded":"2023-11-28T14:32:53.503689+0000","last_fullsized":"2023-11-28T14:32:53.503689+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6252138,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":9888,"num_write":13,"num_write_kb":3687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":77,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-28T14:32:53.488237+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:32:53.488237+0000","last_peered":"2023-11-28T14:32:53.488237+0000","last_clean":"2023-11-28T14:32:53.488237+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:32:53.488237+0000","last_undegraded":"2023-11-28T14:32:53.488237+0000","last_fullsized":"2023-11-28T14:32:53.488237+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":88975499,"num_objects":100,"num_object_clones":55,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":365,"num_read_kb":215262,"num_write":473,"num_write_kb":135581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":628,"ondisk_log_size":628,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":68496,"kb_used_data":41672,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94303344,"statfs":{"total":96636764160,"available":96566624256,"internally_reserved":0,"allocated":42672128,"data_stored":43477472,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705720,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73924,"kb_used_data":47100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94297916,"statfs":{"total":96636764160,"available":96561065984,"internally_reserved":0,"allocated":48230400,"data_stored":48208436,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":53492,"kb_used_data":26668,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94318348,"statfs":{"total":96636764160,"available":96581988352,"internally_reserved":0,"allocated":27308032,"data_stored":28851380,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705719,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":68260,"kb_used_data":41436,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94303580,"statfs":{"total":96636764160,"available":96566865920,"internally_reserved":0,"allocated":42430464,"data_stored":42700009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:32:55.467 INFO:tasks.rados.rados.0.smithi055.stdout:243: done (0 left) 2023-11-28T14:32:55.467 INFO:tasks.rados.rados.0.smithi055.stdout:244: delete oid 29 current snap is 21 2023-11-28T14:32:55.497 INFO:tasks.rados.rados.0.smithi055.stdout:244: done (0 left) 2023-11-28T14:32:55.497 INFO:tasks.rados.rados.0.smithi055.stdout:245: snap_create 2023-11-28T14:32:55.739 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:32:55.849 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:55.847+0000 7fe2eb35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:55.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:55.851+0000 7fe2eb35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:56.468 INFO:tasks.rados.rados.0.smithi055.stdout:245: done (0 left) 2023-11-28T14:32:56.517 INFO:tasks.rados.rados.0.smithi055.stdout:246: write (excl) oid 45 current snap is 22 2023-11-28T14:32:56.517 INFO:tasks.rados.rados.0.smithi055.stdout:246: seq_num 141 ranges {769787=704463,2267961=101826} 2023-11-28T14:32:56.518 INFO:tasks.rados.rados.0.smithi055.stdout:246: writing smithi05583006-45 from 769787 to 1474250 tid 1 2023-11-28T14:32:56.518 INFO:tasks.rados.rados.0.smithi055.stdout:246: writing smithi05583006-45 from 2267961 to 2369787 tid 2 2023-11-28T14:32:56.518 INFO:tasks.rados.rados.0.smithi055.stdout:247: snap_remove snap 20 2023-11-28T14:32:57.476 INFO:tasks.rados.rados.0.smithi055.stdout:247: done (1 left) 2023-11-28T14:32:57.476 INFO:tasks.rados.rados.0.smithi055.stdout:248: write (excl) oid 9 current snap is 22 2023-11-28T14:32:57.476 INFO:tasks.rados.rados.0.smithi055.stdout:246: finishing write tid 1 to smithi05583006-45 2023-11-28T14:32:57.476 INFO:tasks.rados.rados.0.smithi055.stdout:248: seq_num 142 ranges {786594=779712,2326443=60151} 2023-11-28T14:32:57.484 INFO:tasks.rados.rados.0.smithi055.stdout:248: writing smithi05583006-9 from 786594 to 1566306 tid 1 2023-11-28T14:32:57.489 INFO:tasks.rados.rados.0.smithi055.stdout:248: writing smithi05583006-9 from 2326443 to 2386594 tid 2 2023-11-28T14:32:57.489 INFO:tasks.rados.rados.0.smithi055.stdout:249: write (excl) oid 20 current snap is 22 2023-11-28T14:32:57.489 INFO:tasks.rados.rados.0.smithi055.stdout:249: seq_num 143 ranges {403401=454961,1584925=506039} 2023-11-28T14:32:57.493 INFO:tasks.rados.rados.0.smithi055.stdout:249: writing smithi05583006-20 from 403401 to 858362 tid 1 2023-11-28T14:32:57.501 INFO:tasks.rados.rados.0.smithi055.stdout:249: writing smithi05583006-20 from 1584925 to 2090964 tid 2 2023-11-28T14:32:57.501 INFO:tasks.rados.rados.0.smithi055.stdout:250: snap_create 2023-11-28T14:32:57.501 INFO:tasks.rados.rados.0.smithi055.stdout:246: finishing write tid 2 to smithi05583006-45 2023-11-28T14:32:57.501 INFO:tasks.rados.rados.0.smithi055.stdout:246: finishing write tid 3 to smithi05583006-45 2023-11-28T14:32:57.501 INFO:tasks.rados.rados.0.smithi055.stdout:246: finishing write tid 4 to smithi05583006-45 2023-11-28T14:32:57.501 INFO:tasks.rados.rados.0.smithi055.stdout:246: oid 45 updating version 0 to 41 2023-11-28T14:32:57.501 INFO:tasks.rados.rados.0.smithi055.stdout:246: oid 45 updating version 41 to 42 2023-11-28T14:32:57.501 INFO:tasks.rados.rados.0.smithi055.stdout:246: oid 45 updating version 42 to 43 2023-11-28T14:32:57.501 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 45 v 43 (ObjNum 141 snap 22 seq_num 141) dirty exists 2023-11-28T14:32:57.502 INFO:tasks.rados.rados.0.smithi055.stdout:246: left oid 45 (ObjNum 141 snap 22 seq_num 141) 2023-11-28T14:32:57.561 INFO:tasks.rados.rados.0.smithi055.stdout:248: finishing write tid 1 to smithi05583006-9 2023-11-28T14:32:57.562 INFO:tasks.rados.rados.0.smithi055.stdout:249: finishing write tid 1 to smithi05583006-20 2023-11-28T14:32:57.571 INFO:tasks.rados.rados.0.smithi055.stdout:248: finishing write tid 2 to smithi05583006-9 2023-11-28T14:32:57.581 INFO:tasks.rados.rados.0.smithi055.stdout:249: finishing write tid 2 to smithi05583006-20 2023-11-28T14:32:57.582 INFO:tasks.rados.rados.0.smithi055.stdout:248: finishing write tid 3 to smithi05583006-9 2023-11-28T14:32:57.584 INFO:tasks.rados.rados.0.smithi055.stdout:248: finishing write tid 4 to smithi05583006-9 2023-11-28T14:32:57.584 INFO:tasks.rados.rados.0.smithi055.stdout:248: oid 9 updating version 0 to 33 2023-11-28T14:32:57.584 INFO:tasks.rados.rados.0.smithi055.stdout:248: oid 9 updating version 33 to 35 2023-11-28T14:32:57.584 INFO:tasks.rados.rados.0.smithi055.stdout:248: oid 9 version 35 is already newer than 34 2023-11-28T14:32:57.584 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 9 v 35 (ObjNum 142 snap 22 seq_num 142) dirty exists 2023-11-28T14:32:57.584 INFO:tasks.rados.rados.0.smithi055.stdout:248: left oid 9 (ObjNum 142 snap 22 seq_num 142) 2023-11-28T14:32:57.592 INFO:tasks.rados.rados.0.smithi055.stdout:249: finishing write tid 3 to smithi05583006-20 2023-11-28T14:32:57.593 INFO:tasks.rados.rados.0.smithi055.stdout:249: finishing write tid 4 to smithi05583006-20 2023-11-28T14:32:57.593 INFO:tasks.rados.rados.0.smithi055.stdout:249: oid 20 updating version 0 to 18 2023-11-28T14:32:57.593 INFO:tasks.rados.rados.0.smithi055.stdout:249: oid 20 version 18 is already newer than 17 2023-11-28T14:32:57.593 INFO:tasks.rados.rados.0.smithi055.stdout:249: oid 20 updating version 18 to 19 2023-11-28T14:32:57.593 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 20 v 19 (ObjNum 143 snap 22 seq_num 143) dirty exists 2023-11-28T14:32:57.593 INFO:tasks.rados.rados.0.smithi055.stdout:249: left oid 20 (ObjNum 143 snap 22 seq_num 143) 2023-11-28T14:32:57.922 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:32:58.037 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:58.035+0000 7f2c90f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:58.040 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:32:58.039+0000 7f2c90f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:32:58.478 INFO:tasks.rados.rados.0.smithi055.stdout:246: done (3 left) 2023-11-28T14:32:58.478 INFO:tasks.rados.rados.0.smithi055.stdout:248: done (2 left) 2023-11-28T14:32:58.478 INFO:tasks.rados.rados.0.smithi055.stdout:249: done (1 left) 2023-11-28T14:32:58.478 INFO:tasks.rados.rados.0.smithi055.stdout:250: done (0 left) 2023-11-28T14:32:58.478 INFO:tasks.rados.rados.0.smithi055.stdout:251: snap_create 2023-11-28T14:32:59.482 INFO:tasks.rados.rados.0.smithi055.stdout:251: done (0 left) 2023-11-28T14:32:59.482 INFO:tasks.rados.rados.0.smithi055.stdout:252: snap_remove snap 21 2023-11-28T14:33:00.113 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:33:00.237 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:00.235+0000 7fc63af9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:00.240 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:00.239+0000 7fc63af9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:00.370 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:33:00.370 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:33:00.370 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:33:00.485 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:00.481+0000 7fda47a06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:00.487 INFO:tasks.rados.rados.0.smithi055.stdout:252: done (0 left) 2023-11-28T14:33:00.488 INFO:tasks.rados.rados.0.smithi055.stdout:253: write (excl) oid 9 current snap is 24 2023-11-28T14:33:00.488 INFO:tasks.rados.rados.0.smithi055.stdout:253: seq_num 144 ranges {420208=530210,1527054=766050} 2023-11-28T14:33:00.488 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:00.487+0000 7fda47a06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:00.493 INFO:tasks.rados.rados.0.smithi055.stdout:253: writing smithi05583006-9 from 420208 to 950418 tid 1 2023-11-28T14:33:00.500 INFO:tasks.rados.rados.0.smithi055.stdout:253: writing smithi05583006-9 from 1527054 to 2293104 tid 2 2023-11-28T14:33:00.500 INFO:tasks.rados.rados.0.smithi055.stdout:254: snap_create 2023-11-28T14:33:00.569 INFO:tasks.rados.rados.0.smithi055.stdout:253: finishing write tid 1 to smithi05583006-9 2023-11-28T14:33:00.592 INFO:tasks.rados.rados.0.smithi055.stdout:253: finishing write tid 2 to smithi05583006-9 2023-11-28T14:33:00.592 INFO:tasks.rados.rados.0.smithi055.stdout:253: finishing write tid 3 to smithi05583006-9 2023-11-28T14:33:00.595 INFO:tasks.rados.rados.0.smithi055.stdout:253: finishing write tid 4 to smithi05583006-9 2023-11-28T14:33:00.595 INFO:tasks.rados.rados.0.smithi055.stdout:253: oid 9 updating version 0 to 39 2023-11-28T14:33:00.595 INFO:tasks.rados.rados.0.smithi055.stdout:253: oid 9 version 39 is already newer than 39 2023-11-28T14:33:00.595 INFO:tasks.rados.rados.0.smithi055.stdout:253: oid 9 version 39 is already newer than 37 2023-11-28T14:33:00.595 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 9 v 39 (ObjNum 144 snap 24 seq_num 144) dirty exists 2023-11-28T14:33:00.596 INFO:tasks.rados.rados.0.smithi055.stdout:253: left oid 9 (ObjNum 144 snap 24 seq_num 144) 2023-11-28T14:33:00.707 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:33:00.708 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:33:00.719 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2023-11-28T14:33:00.485362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83414670,"num_objects":106,"num_object_clones":59,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":477,"num_read_kb":218577,"num_write":625,"num_write_kb":140508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":753,"ondisk_log_size":753,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":263468,"kb_used_data":152812,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377223892,"statfs":{"total":386547056640,"available":386277265408,"internally_reserved":0,"allocated":156479488,"data_stored":161342971,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6151197,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":9,"num_read_kb":3224,"num_write":15,"num_write_kb":3033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016882"},"pg_stats":[{"pgid":"3.f","version":"60'133","reported_seq":220,"reported_epoch":62,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:32:57.486378+0000","last_change":"2023-11-28T14:32:57.486378+0000","last_active":"2023-11-28T14:32:57.486378+0000","last_peered":"2023-11-28T14:32:57.486378+0000","last_clean":"2023-11-28T14:32:57.486378+0000","last_became_active":"2023-11-28T14:32:09.321992+0000","last_became_peered":"2023-11-28T14:32:09.321992+0000","last_unstale":"2023-11-28T14:32:57.486378+0000","last_undegraded":"2023-11-28T14:32:57.486378+0000","last_fullsized":"2023-11-28T14:32:57.486378+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11764834,"num_objects":19,"num_object_clones":12,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":65,"num_read_kb":38071,"num_write":98,"num_write_kb":28657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'26","reported_seq":126,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.515712+0000","last_change":"2023-11-28T14:32:57.523886+0000","last_active":"2023-11-28T14:32:57.515712+0000","last_peered":"2023-11-28T14:32:57.515712+0000","last_clean":"2023-11-28T14:32:57.515712+0000","last_became_active":"2023-11-28T14:32:09.308848+0000","last_became_peered":"2023-11-28T14:32:09.308848+0000","last_unstale":"2023-11-28T14:32:57.515712+0000","last_undegraded":"2023-11-28T14:32:57.515712+0000","last_fullsized":"2023-11-28T14:32:57.515712+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050454,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":6436,"num_write":19,"num_write_kb":5379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"62'35","reported_seq":140,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.572767+0000","last_change":"2023-11-28T14:32:57.517101+0000","last_active":"2023-11-28T14:32:57.572767+0000","last_peered":"2023-11-28T14:32:57.572767+0000","last_clean":"2023-11-28T14:32:57.572767+0000","last_became_active":"2023-11-28T14:32:09.337560+0000","last_became_peered":"2023-11-28T14:32:09.337560+0000","last_unstale":"2023-11-28T14:32:57.572767+0000","last_undegraded":"2023-11-28T14:32:57.572767+0000","last_fullsized":"2023-11-28T14:32:57.572767+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4571504,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":26,"num_read_kb":24015,"num_write":24,"num_write_kb":6023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"55'40","reported_seq":140,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.485637+0000","last_change":"2023-11-28T14:32:57.491832+0000","last_active":"2023-11-28T14:32:57.485637+0000","last_peered":"2023-11-28T14:32:57.485637+0000","last_clean":"2023-11-28T14:32:57.485637+0000","last_became_active":"2023-11-28T14:32:09.344782+0000","last_became_peered":"2023-11-28T14:32:09.344782+0000","last_unstale":"2023-11-28T14:32:57.485637+0000","last_undegraded":"2023-11-28T14:32:57.485637+0000","last_fullsized":"2023-11-28T14:32:57.485637+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764735,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":28,"num_read_kb":16797,"num_write":26,"num_write_kb":7877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"60'29","reported_seq":139,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-11-28T14:32:56.508391+0000","last_change":"2023-11-28T14:32:56.508391+0000","last_active":"2023-11-28T14:32:56.508391+0000","last_peered":"2023-11-28T14:32:56.508391+0000","last_clean":"2023-11-28T14:32:56.508391+0000","last_became_active":"2023-11-28T14:32:09.326269+0000","last_became_peered":"2023-11-28T14:32:09.326269+0000","last_unstale":"2023-11-28T14:32:56.508391+0000","last_undegraded":"2023-11-28T14:32:56.508391+0000","last_fullsized":"2023-11-28T14:32:56.508391+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8672346,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":22057,"num_write":30,"num_write_kb":8058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"60'34","reported_seq":126,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:56.517995+0000","last_change":"2023-11-28T14:32:55.486286+0000","last_active":"2023-11-28T14:32:56.517995+0000","last_peered":"2023-11-28T14:32:56.517995+0000","last_clean":"2023-11-28T14:32:56.517995+0000","last_became_active":"2023-11-28T14:32:09.308846+0000","last_became_peered":"2023-11-28T14:32:09.308846+0000","last_unstale":"2023-11-28T14:32:56.517995+0000","last_undegraded":"2023-11-28T14:32:56.517995+0000","last_fullsized":"2023-11-28T14:32:56.517995+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4437015,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":15,"num_read_kb":5568,"num_write":21,"num_write_kb":5426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"59'33","reported_seq":136,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.480869+0000","last_change":"2023-11-28T14:32:57.491889+0000","last_active":"2023-11-28T14:32:57.480869+0000","last_peered":"2023-11-28T14:32:57.480869+0000","last_clean":"2023-11-28T14:32:57.480869+0000","last_became_active":"2023-11-28T14:32:09.316660+0000","last_became_peered":"2023-11-28T14:32:09.316660+0000","last_unstale":"2023-11-28T14:32:57.480869+0000","last_undegraded":"2023-11-28T14:32:57.480869+0000","last_fullsized":"2023-11-28T14:32:57.480869+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2285752,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":15,"num_read_kb":13968,"num_write":27,"num_write_kb":7610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"45'29","reported_seq":133,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.509826+0000","last_change":"2023-11-28T14:32:57.518551+0000","last_active":"2023-11-28T14:32:57.509826+0000","last_peered":"2023-11-28T14:32:57.509826+0000","last_clean":"2023-11-28T14:32:57.509826+0000","last_became_active":"2023-11-28T14:32:09.358025+0000","last_became_peered":"2023-11-28T14:32:09.358025+0000","last_unstale":"2023-11-28T14:32:57.509826+0000","last_undegraded":"2023-11-28T14:32:57.509826+0000","last_fullsized":"2023-11-28T14:32:57.509826+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5680700,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":19,"num_read_kb":10485,"num_write":21,"num_write_kb":5738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":220,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-11-28T14:32:56.485674+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:32:56.485674+0000","last_peered":"2023-11-28T14:32:56.485674+0000","last_clean":"2023-11-28T14:32:56.485674+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:32:56.485674+0000","last_undegraded":"2023-11-28T14:32:56.485674+0000","last_fullsized":"2023-11-28T14:32:56.485674+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":85,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.512215+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:32:57.512215+0000","last_peered":"2023-11-28T14:32:57.512215+0000","last_clean":"2023-11-28T14:32:57.512215+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:32:57.512215+0000","last_undegraded":"2023-11-28T14:32:57.512215+0000","last_fullsized":"2023-11-28T14:32:57.512215+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"54'97","reported_seq":205,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.514229+0000","last_change":"2023-11-28T14:32:57.517983+0000","last_active":"2023-11-28T14:32:57.514229+0000","last_peered":"2023-11-28T14:32:57.514229+0000","last_clean":"2023-11-28T14:32:57.514229+0000","last_became_active":"2023-11-28T14:32:09.335805+0000","last_became_peered":"2023-11-28T14:32:09.335805+0000","last_unstale":"2023-11-28T14:32:57.514229+0000","last_undegraded":"2023-11-28T14:32:57.514229+0000","last_fullsized":"2023-11-28T14:32:57.514229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15126135,"num_objects":15,"num_object_clones":8,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":70,"num_read_kb":37491,"num_write":76,"num_write_kb":21255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":85,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.487407+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:32:57.487407+0000","last_peered":"2023-11-28T14:32:57.487407+0000","last_clean":"2023-11-28T14:32:57.487407+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:32:57.487407+0000","last_undegraded":"2023-11-28T14:32:57.487407+0000","last_fullsized":"2023-11-28T14:32:57.487407+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'43","reported_seq":139,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-11-28T14:32:56.566862+0000","last_change":"2023-11-28T14:32:55.653749+0000","last_active":"2023-11-28T14:32:56.566862+0000","last_peered":"2023-11-28T14:32:56.566862+0000","last_clean":"2023-11-28T14:32:56.566862+0000","last_became_active":"2023-11-28T14:32:09.350412+0000","last_became_peered":"2023-11-28T14:32:09.350412+0000","last_unstale":"2023-11-28T14:32:56.566862+0000","last_undegraded":"2023-11-28T14:32:56.566862+0000","last_fullsized":"2023-11-28T14:32:56.566862+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4521050,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":22,"num_read_kb":8139,"num_write":30,"num_write_kb":8770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":85,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.511356+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:32:57.511356+0000","last_peered":"2023-11-28T14:32:57.511356+0000","last_clean":"2023-11-28T14:32:57.511356+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:32:57.511356+0000","last_undegraded":"2023-11-28T14:32:57.511356+0000","last_fullsized":"2023-11-28T14:32:57.511356+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'32","reported_seq":120,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-11-28T14:32:56.482294+0000","last_change":"2023-11-28T14:32:55.653315+0000","last_active":"2023-11-28T14:32:56.482294+0000","last_peered":"2023-11-28T14:32:56.482294+0000","last_clean":"2023-11-28T14:32:56.482294+0000","last_became_active":"2023-11-28T14:32:09.343664+0000","last_became_peered":"2023-11-28T14:32:09.343664+0000","last_unstale":"2023-11-28T14:32:56.482294+0000","last_undegraded":"2023-11-28T14:32:56.482294+0000","last_fullsized":"2023-11-28T14:32:56.482294+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932805,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":19,"num_write_kb":5527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":85,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.485334+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:32:57.485334+0000","last_peered":"2023-11-28T14:32:57.485334+0000","last_clean":"2023-11-28T14:32:57.485334+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:32:57.485334+0000","last_undegraded":"2023-11-28T14:32:57.485334+0000","last_fullsized":"2023-11-28T14:32:57.485334+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'11","reported_seq":115,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.517262+0000","last_change":"2023-11-28T14:32:57.527961+0000","last_active":"2023-11-28T14:32:57.517262+0000","last_peered":"2023-11-28T14:32:57.517262+0000","last_clean":"2023-11-28T14:32:57.517262+0000","last_became_active":"2023-11-28T14:32:09.311653+0000","last_became_peered":"2023-11-28T14:32:09.311653+0000","last_unstale":"2023-11-28T14:32:57.517262+0000","last_undegraded":"2023-11-28T14:32:57.517262+0000","last_fullsized":"2023-11-28T14:32:57.517262+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2100875,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5912,"num_write":8,"num_write_kb":1910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"55'32","reported_seq":125,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.514091+0000","last_change":"2023-11-28T14:32:57.526616+0000","last_active":"2023-11-28T14:32:57.514091+0000","last_peered":"2023-11-28T14:32:57.514091+0000","last_clean":"2023-11-28T14:32:57.514091+0000","last_became_active":"2023-11-28T14:32:09.316944+0000","last_became_peered":"2023-11-28T14:32:09.316944+0000","last_unstale":"2023-11-28T14:32:57.514091+0000","last_undegraded":"2023-11-28T14:32:57.514091+0000","last_fullsized":"2023-11-28T14:32:57.514091+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":9,"num_read_kb":5,"num_write":21,"num_write_kb":6458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":89,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.512996+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:32:57.512996+0000","last_peered":"2023-11-28T14:32:57.512996+0000","last_clean":"2023-11-28T14:32:57.512996+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:32:57.512996+0000","last_undegraded":"2023-11-28T14:32:57.512996+0000","last_fullsized":"2023-11-28T14:32:57.512996+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"55'29","reported_seq":130,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.518569+0000","last_change":"2023-11-28T14:32:57.524394+0000","last_active":"2023-11-28T14:32:57.518569+0000","last_peered":"2023-11-28T14:32:57.518569+0000","last_clean":"2023-11-28T14:32:57.518569+0000","last_became_active":"2023-11-28T14:32:09.309510+0000","last_became_peered":"2023-11-28T14:32:09.309510+0000","last_unstale":"2023-11-28T14:32:57.518569+0000","last_undegraded":"2023-11-28T14:32:57.518569+0000","last_fullsized":"2023-11-28T14:32:57.518569+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252138,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":7924,"num_write":21,"num_write_kb":6086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":87,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-11-28T14:32:56.485872+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:32:56.485872+0000","last_peered":"2023-11-28T14:32:56.485872+0000","last_clean":"2023-11-28T14:32:56.485872+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:32:56.485872+0000","last_undegraded":"2023-11-28T14:32:56.485872+0000","last_fullsized":"2023-11-28T14:32:56.485872+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"53'34","reported_seq":139,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-11-28T14:32:56.486500+0000","last_change":"2023-11-28T14:32:55.668764+0000","last_active":"2023-11-28T14:32:56.486500+0000","last_peered":"2023-11-28T14:32:56.486500+0000","last_clean":"2023-11-28T14:32:56.486500+0000","last_became_active":"2023-11-28T14:32:09.356991+0000","last_became_peered":"2023-11-28T14:32:09.356991+0000","last_unstale":"2023-11-28T14:32:56.486500+0000","last_undegraded":"2023-11-28T14:32:56.486500+0000","last_fullsized":"2023-11-28T14:32:56.486500+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4823576,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":19,"num_read_kb":11725,"num_write":30,"num_write_kb":9213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":89,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.520071+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:32:57.520071+0000","last_peered":"2023-11-28T14:32:57.520071+0000","last_clean":"2023-11-28T14:32:57.520071+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:32:57.520071+0000","last_undegraded":"2023-11-28T14:32:57.520071+0000","last_fullsized":"2023-11-28T14:32:57.520071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"62'19","reported_seq":127,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.582675+0000","last_change":"2023-11-28T14:32:57.523502+0000","last_active":"2023-11-28T14:32:57.582675+0000","last_peered":"2023-11-28T14:32:57.582675+0000","last_clean":"2023-11-28T14:32:57.582675+0000","last_became_active":"2023-11-28T14:32:09.337548+0000","last_became_peered":"2023-11-28T14:32:09.337548+0000","last_unstale":"2023-11-28T14:32:57.582675+0000","last_undegraded":"2023-11-28T14:32:57.582675+0000","last_fullsized":"2023-11-28T14:32:57.582675+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4840383,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":9889,"num_write":17,"num_write_kb":4627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":85,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T14:32:57.510434+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:32:57.510434+0000","last_peered":"2023-11-28T14:32:57.510434+0000","last_clean":"2023-11-28T14:32:57.510434+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:32:57.510434+0000","last_undegraded":"2023-11-28T14:32:57.510434+0000","last_fullsized":"2023-11-28T14:32:57.510434+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":82824302,"num_objects":104,"num_object_clones":59,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":374,"num_read_kb":218486,"num_write":488,"num_write_kb":138614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":656,"ondisk_log_size":656,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":68736,"kb_used_data":41912,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94303104,"statfs":{"total":96636764160,"available":96566378496,"internally_reserved":0,"allocated":42917888,"data_stored":44422984,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73160,"kb_used_data":45888,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94298680,"statfs":{"total":96636764160,"available":96561848320,"internally_reserved":0,"allocated":46989312,"data_stored":47729297,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":53084,"kb_used_data":26260,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94318756,"statfs":{"total":96636764160,"available":96582406144,"internally_reserved":0,"allocated":26890240,"data_stored":29037202,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705724,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":68488,"kb_used_data":38752,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94303352,"statfs":{"total":96636764160,"available":96566632448,"internally_reserved":0,"allocated":39682048,"data_stored":40153488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:33:01.501 INFO:tasks.rados.rados.0.smithi055.stdout:253: done (1 left) 2023-11-28T14:33:01.501 INFO:tasks.rados.rados.0.smithi055.stdout:254: done (0 left) 2023-11-28T14:33:01.501 INFO:tasks.rados.rados.0.smithi055.stdout:255: read oid 33 snap -1 2023-11-28T14:33:01.501 INFO:tasks.rados.rados.0.smithi055.stdout:255: expect (ObjNum 103 snap 11 seq_num 103) 2023-11-28T14:33:01.501 INFO:tasks.rados.rados.0.smithi055.stdout:256: read oid 32 snap -1 2023-11-28T14:33:01.501 INFO:tasks.rados.rados.0.smithi055.stdout:256: expect (ObjNum 31 snap 0 seq_num 31) 2023-11-28T14:33:01.501 INFO:tasks.rados.rados.0.smithi055.stdout:257: read oid 37 snap -1 2023-11-28T14:33:01.501 INFO:tasks.rados.rados.0.smithi055.stdout:257: expect deleted 2023-11-28T14:33:01.501 INFO:tasks.rados.rados.0.smithi055.stdout:258: delete oid 45 current snap is 25 2023-11-28T14:33:01.589 INFO:tasks.rados.rados.0.smithi055.stdout:255: expect (ObjNum 103 snap 11 seq_num 103) 2023-11-28T14:33:01.819 INFO:tasks.rados.rados.0.smithi055.stdout:256: expect (ObjNum 31 snap 0 seq_num 31) 2023-11-28T14:33:01.915 INFO:tasks.rados.rados.0.smithi055.stdout:255: done (3 left) 2023-11-28T14:33:01.915 INFO:tasks.rados.rados.0.smithi055.stdout:256: done (2 left) 2023-11-28T14:33:01.915 INFO:tasks.rados.rados.0.smithi055.stdout:257: done (1 left) 2023-11-28T14:33:01.915 INFO:tasks.rados.rados.0.smithi055.stdout:258: done (0 left) 2023-11-28T14:33:01.915 INFO:tasks.rados.rados.0.smithi055.stdout:259: snap_create 2023-11-28T14:33:02.312 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:33:02.425 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:02.423+0000 7fd7bfb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:02.428 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:02.427+0000 7fd7bfb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:02.514 INFO:tasks.rados.rados.0.smithi055.stdout:259: done (0 left) 2023-11-28T14:33:02.514 INFO:tasks.rados.rados.0.smithi055.stdout:260: snap_remove snap 24 2023-11-28T14:33:03.511 INFO:tasks.rados.rados.0.smithi055.stdout:260: done (0 left) 2023-11-28T14:33:03.511 INFO:tasks.rados.rados.0.smithi055.stdout:261: delete oid 21 current snap is 26 2023-11-28T14:33:03.543 INFO:tasks.rados.rados.0.smithi055.stdout:261: done (0 left) 2023-11-28T14:33:03.543 INFO:tasks.rados.rados.0.smithi055.stdout:262: write (excl) oid 37 current snap is 26 2023-11-28T14:33:03.543 INFO:tasks.rados.rados.0.smithi055.stdout:262: seq_num 147 ranges {470629=472310,1418853=513377} 2023-11-28T14:33:03.548 INFO:tasks.rados.rados.0.smithi055.stdout:262: writing smithi05583006-37 from 470629 to 942939 tid 1 2023-11-28T14:33:03.553 INFO:tasks.rados.rados.0.smithi055.stdout:262: writing smithi05583006-37 from 1418853 to 1932230 tid 2 2023-11-28T14:33:03.553 INFO:tasks.rados.rados.0.smithi055.stdout:263: write (excl) oid 50 current snap is 26 2023-11-28T14:33:03.553 INFO:tasks.rados.rados.0.smithi055.stdout:263: seq_num 148 ranges {487436=547559,1760982=657035} 2023-11-28T14:33:03.559 INFO:tasks.rados.rados.0.smithi055.stdout:263: writing smithi05583006-50 from 487436 to 1034995 tid 1 2023-11-28T14:33:03.567 INFO:tasks.rados.rados.0.smithi055.stdout:263: writing smithi05583006-50 from 1760982 to 2418017 tid 2 2023-11-28T14:33:03.567 INFO:tasks.rados.rados.0.smithi055.stdout:264: snap_remove snap 22 2023-11-28T14:33:04.505 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:33:04.521 INFO:tasks.rados.rados.0.smithi055.stdout:264: done (2 left) 2023-11-28T14:33:04.521 INFO:tasks.rados.rados.0.smithi055.stdout:265: write (excl) oid 6 current snap is 26 2023-11-28T14:33:04.521 INFO:tasks.rados.rados.0.smithi055.stdout:262: finishing write tid 1 to smithi05583006-37 2023-11-28T14:33:04.521 INFO:tasks.rados.rados.0.smithi055.stdout:265: seq_num 149 ranges {504243=622808,1819464=517046} 2023-11-28T14:33:04.527 INFO:tasks.rados.rados.0.smithi055.stdout:265: writing smithi05583006-6 from 504243 to 1127051 tid 1 2023-11-28T14:33:04.534 INFO:tasks.rados.rados.0.smithi055.stdout:265: writing smithi05583006-6 from 1819464 to 2336510 tid 2 2023-11-28T14:33:04.534 INFO:tasks.rados.rados.0.smithi055.stdout:266: write oid 7 current snap is 26 2023-11-28T14:33:04.534 INFO:tasks.rados.rados.0.smithi055.stdout:263: finishing write tid 1 to smithi05583006-50 2023-11-28T14:33:04.534 INFO:tasks.rados.rados.0.smithi055.stdout:266: seq_num 150 ranges {521050=698057,1877946=643104} 2023-11-28T14:33:04.541 INFO:tasks.rados.rados.0.smithi055.stdout:266: writing smithi05583006-7 from 521050 to 1219107 tid 1 2023-11-28T14:33:04.549 INFO:tasks.rados.rados.0.smithi055.stdout:266: writing smithi05583006-7 from 1877946 to 2521050 tid 2 2023-11-28T14:33:04.549 INFO:tasks.rados.rados.0.smithi055.stdout:267: snap_create 2023-11-28T14:33:04.549 INFO:tasks.rados.rados.0.smithi055.stdout:262: finishing write tid 2 to smithi05583006-37 2023-11-28T14:33:04.549 INFO:tasks.rados.rados.0.smithi055.stdout:262: finishing write tid 3 to smithi05583006-37 2023-11-28T14:33:04.549 INFO:tasks.rados.rados.0.smithi055.stdout:263: finishing write tid 2 to smithi05583006-50 2023-11-28T14:33:04.549 INFO:tasks.rados.rados.0.smithi055.stdout:262: finishing write tid 4 to smithi05583006-37 2023-11-28T14:33:04.549 INFO:tasks.rados.rados.0.smithi055.stdout:262: oid 37 updating version 0 to 54 2023-11-28T14:33:04.550 INFO:tasks.rados.rados.0.smithi055.stdout:262: oid 37 version 54 is already newer than 52 2023-11-28T14:33:04.550 INFO:tasks.rados.rados.0.smithi055.stdout:262: oid 37 version 54 is already newer than 54 2023-11-28T14:33:04.550 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 37 v 54 (ObjNum 147 snap 26 seq_num 147) dirty exists 2023-11-28T14:33:04.550 INFO:tasks.rados.rados.0.smithi055.stdout:262: left oid 37 (ObjNum 147 snap 26 seq_num 147) 2023-11-28T14:33:04.550 INFO:tasks.rados.rados.0.smithi055.stdout:263: finishing write tid 3 to smithi05583006-50 2023-11-28T14:33:04.551 INFO:tasks.rados.rados.0.smithi055.stdout:263: finishing write tid 4 to smithi05583006-50 2023-11-28T14:33:04.551 INFO:tasks.rados.rados.0.smithi055.stdout:263: oid 50 updating version 0 to 36 2023-11-28T14:33:04.552 INFO:tasks.rados.rados.0.smithi055.stdout:263: oid 50 updating version 36 to 37 2023-11-28T14:33:04.552 INFO:tasks.rados.rados.0.smithi055.stdout:263: oid 50 updating version 37 to 38 2023-11-28T14:33:04.553 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 50 v 38 (ObjNum 148 snap 26 seq_num 148) dirty exists 2023-11-28T14:33:04.553 INFO:tasks.rados.rados.0.smithi055.stdout:263: left oid 50 (ObjNum 148 snap 26 seq_num 148) 2023-11-28T14:33:04.628 INFO:tasks.rados.rados.0.smithi055.stdout:265: finishing write tid 1 to smithi05583006-6 2023-11-28T14:33:04.629 INFO:tasks.rados.rados.0.smithi055.stdout:266: finishing write tid 1 to smithi05583006-7 2023-11-28T14:33:04.637 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:04.634+0000 7fd46fb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:04.640 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:04.639+0000 7fd46fb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:04.653 INFO:tasks.rados.rados.0.smithi055.stdout:266: finishing write tid 2 to smithi05583006-7 2023-11-28T14:33:04.668 INFO:tasks.rados.rados.0.smithi055.stdout:266: finishing write tid 3 to smithi05583006-7 2023-11-28T14:33:04.669 INFO:tasks.rados.rados.0.smithi055.stdout:266: finishing write tid 4 to smithi05583006-7 2023-11-28T14:33:04.669 INFO:tasks.rados.rados.0.smithi055.stdout:266: oid 7 updating version 0 to 33 2023-11-28T14:33:04.669 INFO:tasks.rados.rados.0.smithi055.stdout:266: oid 7 updating version 33 to 34 2023-11-28T14:33:04.669 INFO:tasks.rados.rados.0.smithi055.stdout:266: oid 7 updating version 34 to 35 2023-11-28T14:33:04.670 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 7 v 35 (ObjNum 150 snap 26 seq_num 150) dirty exists 2023-11-28T14:33:04.670 INFO:tasks.rados.rados.0.smithi055.stdout:266: left oid 7 (ObjNum 150 snap 26 seq_num 150) 2023-11-28T14:33:04.672 INFO:tasks.rados.rados.0.smithi055.stdout:265: finishing write tid 2 to smithi05583006-6 2023-11-28T14:33:04.687 INFO:tasks.rados.rados.0.smithi055.stdout:265: finishing write tid 3 to smithi05583006-6 2023-11-28T14:33:04.689 INFO:tasks.rados.rados.0.smithi055.stdout:265: finishing write tid 4 to smithi05583006-6 2023-11-28T14:33:04.689 INFO:tasks.rados.rados.0.smithi055.stdout:265: oid 6 updating version 0 to 32 2023-11-28T14:33:04.689 INFO:tasks.rados.rados.0.smithi055.stdout:265: oid 6 updating version 32 to 33 2023-11-28T14:33:04.689 INFO:tasks.rados.rados.0.smithi055.stdout:265: oid 6 version 33 is already newer than 31 2023-11-28T14:33:04.689 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 6 v 33 (ObjNum 149 snap 26 seq_num 149) dirty exists 2023-11-28T14:33:04.689 INFO:tasks.rados.rados.0.smithi055.stdout:265: left oid 6 (ObjNum 149 snap 26 seq_num 149) 2023-11-28T14:33:05.121 INFO:tasks.rados.rados.0.smithi055.stdout:262: done (4 left) 2023-11-28T14:33:05.121 INFO:tasks.rados.rados.0.smithi055.stdout:263: done (3 left) 2023-11-28T14:33:05.122 INFO:tasks.rados.rados.0.smithi055.stdout:265: done (2 left) 2023-11-28T14:33:05.122 INFO:tasks.rados.rados.0.smithi055.stdout:266: done (1 left) 2023-11-28T14:33:05.122 INFO:tasks.rados.rados.0.smithi055.stdout:267: done (0 left) 2023-11-28T14:33:05.122 INFO:tasks.rados.rados.0.smithi055.stdout:268: read oid 13 snap -1 2023-11-28T14:33:05.122 INFO:tasks.rados.rados.0.smithi055.stdout:268: expect deleted 2023-11-28T14:33:05.122 INFO:tasks.rados.rados.0.smithi055.stdout:269: snap_create 2023-11-28T14:33:05.721 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:33:05.722 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:33:05.722 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-11-28T14:33:05.722 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-11-28T14:33:05.840 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:05.834+0000 7efed426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:05.843 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:05.842+0000 7efed426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:05.855 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:33:05.855 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:33:05.855 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:33:05.855 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:33:05.855 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:33:05.855 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:33:05.856 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:33:05.856 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:33:05.856 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:33:05.856 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:33:05.856 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:33:05.856 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:33:05.858 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:33:05.858 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:33:05.859 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-11-28T14:33:05.971 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:05.965+0000 7f0ca6d15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:05.974 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:05.973+0000 7f0ca6d15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:05.986 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:33:05.986 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:33:05.986 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:33:05.986 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:33:05.986 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:33:05.987 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:33:05.987 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:33:05.987 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:33:05.987 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:33:05.987 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:33:05.987 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:33:05.987 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:33:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:33:05.989 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:33:05.989 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-11-28T14:33:06.102 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:06.097+0000 7f55cdf55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:06.105 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:06.104+0000 7f55cdf55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:06.120 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:33:06.120 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:33:06.120 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:33:06.120 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:33:06.120 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:33:06.120 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:33:06.120 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:33:06.121 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:33:06.121 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:33:06.121 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:33:06.121 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:33:06.121 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:33:06.123 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:33:06.123 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:33:06.123 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-11-28T14:33:06.127 INFO:tasks.rados.rados.0.smithi055.stdout:268: done (1 left) 2023-11-28T14:33:06.127 INFO:tasks.rados.rados.0.smithi055.stdout:269: done (0 left) 2023-11-28T14:33:06.127 INFO:tasks.rados.rados.0.smithi055.stdout:270: write oid 20 current snap is 28 2023-11-28T14:33:06.127 INFO:tasks.rados.rados.0.smithi055.stdout:270: seq_num 151 ranges {537857=773306,1936428=520715} 2023-11-28T14:33:06.134 INFO:tasks.rados.rados.0.smithi055.stdout:270: writing smithi05583006-20 from 537857 to 1311163 tid 1 2023-11-28T14:33:06.141 INFO:tasks.rados.rados.0.smithi055.stdout:270: writing smithi05583006-20 from 1936428 to 2457143 tid 2 2023-11-28T14:33:06.141 INFO:tasks.rados.rados.0.smithi055.stdout:271: delete oid 37 current snap is 28 2023-11-28T14:33:06.212 INFO:tasks.rados.rados.0.smithi055.stdout:271: done (1 left) 2023-11-28T14:33:06.212 INFO:tasks.rados.rados.0.smithi055.stdout:272: read oid 7 snap 26 2023-11-28T14:33:06.212 INFO:tasks.rados.rados.0.smithi055.stdout:272: expect (ObjNum 150 snap 26 seq_num 150) 2023-11-28T14:33:06.212 INFO:tasks.rados.rados.0.smithi055.stdout:273: write (excl) oid 35 current snap is 28 2023-11-28T14:33:06.212 INFO:tasks.rados.rados.0.smithi055.stdout:273: seq_num 153 ranges {571471=640157,1653392=524384} 2023-11-28T14:33:06.218 INFO:tasks.rados.rados.0.smithi055.stdout:273: writing smithi05583006-35 from 571471 to 1211628 tid 1 2023-11-28T14:33:06.224 INFO:tasks.rados.rados.0.smithi055.stdout:273: writing smithi05583006-35 from 1653392 to 2177776 tid 2 2023-11-28T14:33:06.225 INFO:tasks.rados.rados.0.smithi055.stdout:274: read oid 1 snap -1 2023-11-28T14:33:06.225 INFO:tasks.rados.rados.0.smithi055.stdout:274: expect (ObjNum 125 snap 18 seq_num 125) 2023-11-28T14:33:06.225 INFO:tasks.rados.rados.0.smithi055.stdout:275: read oid 6 snap -1 2023-11-28T14:33:06.225 INFO:tasks.rados.rados.0.smithi055.stdout:275: expect (ObjNum 149 snap 26 seq_num 149) 2023-11-28T14:33:06.225 INFO:tasks.rados.rados.0.smithi055.stdout:276: read oid 42 snap -1 2023-11-28T14:33:06.225 INFO:tasks.rados.rados.0.smithi055.stdout:276: expect (ObjNum 87 snap 5 seq_num 87) 2023-11-28T14:33:06.225 INFO:tasks.rados.rados.0.smithi055.stdout:277: read oid 38 snap -1 2023-11-28T14:33:06.225 INFO:tasks.rados.rados.0.smithi055.stdout:277: expect (ObjNum 95 snap 9 seq_num 95) 2023-11-28T14:33:06.225 INFO:tasks.rados.rados.0.smithi055.stdout:278: write oid 10 current snap is 28 2023-11-28T14:33:06.225 INFO:tasks.rados.rados.0.smithi055.stdout:270: finishing write tid 1 to smithi05583006-20 2023-11-28T14:33:06.225 INFO:tasks.rados.rados.0.smithi055.stdout:278: seq_num 154 ranges {588278=715406,1711874=668042} 2023-11-28T14:33:06.232 INFO:tasks.rados.rados.0.smithi055.stdout:278: writing smithi05583006-10 from 588278 to 1303684 tid 1 2023-11-28T14:33:06.240 INFO:tasks.rados.rados.0.smithi055.stdout:278: writing smithi05583006-10 from 1711874 to 2379916 tid 2 2023-11-28T14:33:06.240 INFO:tasks.rados.rados.0.smithi055.stdout:279: write oid 2 current snap is 28 2023-11-28T14:33:06.240 INFO:tasks.rados.rados.0.smithi055.stdout:279: seq_num 155 ranges {605085=790655,2170356=434729} 2023-11-28T14:33:06.248 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:06.240+0000 7f8fd242d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:06.250 INFO:tasks.rados.rados.0.smithi055.stdout:279: writing smithi05583006-2 from 605085 to 1395740 tid 1 2023-11-28T14:33:06.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:06.252+0000 7f8fd242d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:06.258 INFO:tasks.rados.rados.0.smithi055.stdout:279: writing smithi05583006-2 from 2170356 to 2605085 tid 2 2023-11-28T14:33:06.258 INFO:tasks.rados.rados.0.smithi055.stdout:280: write (excl) oid 5 current snap is 28 2023-11-28T14:33:06.258 INFO:tasks.rados.rados.0.smithi055.stdout:280: seq_num 156 ranges {621892=465904,1712485=671711} 2023-11-28T14:33:06.265 INFO:tasks.rados.rados.0.smithi055.stdout:280: writing smithi05583006-5 from 621892 to 1087796 tid 1 2023-11-28T14:33:06.274 INFO:tasks.rados.rados.0.smithi055.stdout:280: writing smithi05583006-5 from 1712485 to 2384196 tid 2 2023-11-28T14:33:06.274 INFO:tasks.rados.rados.0.smithi055.stdout:281: snap_remove snap 25 2023-11-28T14:33:06.356 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:33:06.356 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:33:06.356 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:33:06.356 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:33:06.356 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:33:06.356 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:33:06.356 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:33:06.357 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:33:06.357 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:33:06.357 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:33:06.357 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:33:06.357 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:33:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:33:06.359 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:33:06.710 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:33:06.824 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:06.822+0000 7f624a4de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:06.827 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:06.826+0000 7f624a4de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:07.131 INFO:tasks.rados.rados.0.smithi055.stdout:281: done (10 left) 2023-11-28T14:33:07.131 INFO:tasks.rados.rados.0.smithi055.stdout:282: read oid 47 snap -1 2023-11-28T14:33:07.131 INFO:tasks.rados.rados.0.smithi055.stdout:282: expect (ObjNum 55 snap 1 seq_num 55) 2023-11-28T14:33:07.131 INFO:tasks.rados.rados.0.smithi055.stdout:283: snap_create 2023-11-28T14:33:07.131 INFO:tasks.rados.rados.0.smithi055.stdout:276: expect (ObjNum 87 snap 5 seq_num 87) 2023-11-28T14:33:07.334 INFO:tasks.rados.rados.0.smithi055.stdout:272: expect (ObjNum 150 snap 26 seq_num 150) 2023-11-28T14:33:07.487 INFO:tasks.rados.rados.0.smithi055.stdout:270: finishing write tid 2 to smithi05583006-20 2023-11-28T14:33:07.487 INFO:tasks.rados.rados.0.smithi055.stdout:270: finishing write tid 3 to smithi05583006-20 2023-11-28T14:33:07.487 INFO:tasks.rados.rados.0.smithi055.stdout:270: finishing write tid 4 to smithi05583006-20 2023-11-28T14:33:07.488 INFO:tasks.rados.rados.0.smithi055.stdout:270: oid 20 updating version 0 to 25 2023-11-28T14:33:07.488 INFO:tasks.rados.rados.0.smithi055.stdout:270: oid 20 version 25 is already newer than 23 2023-11-28T14:33:07.488 INFO:tasks.rados.rados.0.smithi055.stdout:270: oid 20 version 25 is already newer than 24 2023-11-28T14:33:07.488 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 20 v 25 (ObjNum 151 snap 28 seq_num 151) dirty exists 2023-11-28T14:33:07.488 INFO:tasks.rados.rados.0.smithi055.stdout:270: left oid 20 (ObjNum 151 snap 28 seq_num 151) 2023-11-28T14:33:07.488 INFO:tasks.rados.rados.0.smithi055.stdout:279: finishing write tid 1 to smithi05583006-2 2023-11-28T14:33:07.488 INFO:tasks.rados.rados.0.smithi055.stdout:275: expect (ObjNum 149 snap 26 seq_num 149) 2023-11-28T14:33:07.620 INFO:tasks.rados.rados.0.smithi055.stdout:277: expect (ObjNum 95 snap 9 seq_num 95) 2023-11-28T14:33:07.797 INFO:tasks.rados.rados.0.smithi055.stdout:280: finishing write tid 1 to smithi05583006-5 2023-11-28T14:33:07.797 INFO:tasks.rados.rados.0.smithi055.stdout:274: expect (ObjNum 125 snap 18 seq_num 125) 2023-11-28T14:33:07.902 INFO:tasks.rados.rados.0.smithi055.stdout:273: finishing write tid 1 to smithi05583006-35 2023-11-28T14:33:07.902 INFO:tasks.rados.rados.0.smithi055.stdout:273: finishing write tid 2 to smithi05583006-35 2023-11-28T14:33:07.902 INFO:tasks.rados.rados.0.smithi055.stdout:279: finishing write tid 2 to smithi05583006-2 2023-11-28T14:33:07.902 INFO:tasks.rados.rados.0.smithi055.stdout:278: finishing write tid 1 to smithi05583006-10 2023-11-28T14:33:07.902 INFO:tasks.rados.rados.0.smithi055.stdout:280: finishing write tid 2 to smithi05583006-5 2023-11-28T14:33:07.902 INFO:tasks.rados.rados.0.smithi055.stdout:273: finishing write tid 3 to smithi05583006-35 2023-11-28T14:33:07.902 INFO:tasks.rados.rados.0.smithi055.stdout:273: finishing write tid 4 to smithi05583006-35 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:273: oid 35 updating version 0 to 38 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:273: oid 35 updating version 38 to 39 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:273: oid 35 version 39 is already newer than 37 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 35 v 39 (ObjNum 153 snap 28 seq_num 153) dirty exists 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:273: left oid 35 (ObjNum 153 snap 28 seq_num 153) 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:279: finishing write tid 3 to smithi05583006-2 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:279: finishing write tid 4 to smithi05583006-2 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:279: oid 2 updating version 0 to 137 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:279: oid 2 version 137 is already newer than 136 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:279: oid 2 updating version 137 to 138 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 2 v 138 (ObjNum 155 snap 28 seq_num 155) dirty exists 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:279: left oid 2 (ObjNum 155 snap 28 seq_num 155) 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:280: finishing write tid 3 to smithi05583006-5 2023-11-28T14:33:07.903 INFO:tasks.rados.rados.0.smithi055.stdout:280: finishing write tid 4 to smithi05583006-5 2023-11-28T14:33:07.904 INFO:tasks.rados.rados.0.smithi055.stdout:280: oid 5 updating version 0 to 100 2023-11-28T14:33:07.904 INFO:tasks.rados.rados.0.smithi055.stdout:280: oid 5 version 100 is already newer than 99 2023-11-28T14:33:07.904 INFO:tasks.rados.rados.0.smithi055.stdout:280: oid 5 updating version 100 to 101 2023-11-28T14:33:07.904 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 5 v 101 (ObjNum 156 snap 28 seq_num 156) dirty exists 2023-11-28T14:33:07.904 INFO:tasks.rados.rados.0.smithi055.stdout:280: left oid 5 (ObjNum 156 snap 28 seq_num 156) 2023-11-28T14:33:07.904 INFO:tasks.rados.rados.0.smithi055.stdout:278: finishing write tid 2 to smithi05583006-10 2023-11-28T14:33:07.904 INFO:tasks.rados.rados.0.smithi055.stdout:278: finishing write tid 3 to smithi05583006-10 2023-11-28T14:33:07.904 INFO:tasks.rados.rados.0.smithi055.stdout:278: finishing write tid 4 to smithi05583006-10 2023-11-28T14:33:07.904 INFO:tasks.rados.rados.0.smithi055.stdout:278: oid 10 updating version 0 to 60 2023-11-28T14:33:07.904 INFO:tasks.rados.rados.0.smithi055.stdout:278: oid 10 updating version 60 to 61 2023-11-28T14:33:07.904 INFO:tasks.rados.rados.0.smithi055.stdout:278: oid 10 updating version 61 to 62 2023-11-28T14:33:07.904 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 10 v 62 (ObjNum 154 snap 28 seq_num 154) dirty exists 2023-11-28T14:33:07.904 INFO:tasks.rados.rados.0.smithi055.stdout:278: left oid 10 (ObjNum 154 snap 28 seq_num 154) 2023-11-28T14:33:07.905 INFO:tasks.rados.rados.0.smithi055.stdout:282: expect (ObjNum 55 snap 1 seq_num 55) 2023-11-28T14:33:08.132 INFO:tasks.rados.rados.0.smithi055.stdout:270: done (11 left) 2023-11-28T14:33:08.133 INFO:tasks.rados.rados.0.smithi055.stdout:272: done (10 left) 2023-11-28T14:33:08.133 INFO:tasks.rados.rados.0.smithi055.stdout:273: done (9 left) 2023-11-28T14:33:08.133 INFO:tasks.rados.rados.0.smithi055.stdout:274: done (8 left) 2023-11-28T14:33:08.133 INFO:tasks.rados.rados.0.smithi055.stdout:275: done (7 left) 2023-11-28T14:33:08.133 INFO:tasks.rados.rados.0.smithi055.stdout:276: done (6 left) 2023-11-28T14:33:08.133 INFO:tasks.rados.rados.0.smithi055.stdout:277: done (5 left) 2023-11-28T14:33:08.133 INFO:tasks.rados.rados.0.smithi055.stdout:278: done (4 left) 2023-11-28T14:33:08.133 INFO:tasks.rados.rados.0.smithi055.stdout:279: done (3 left) 2023-11-28T14:33:08.133 INFO:tasks.rados.rados.0.smithi055.stdout:280: done (2 left) 2023-11-28T14:33:08.133 INFO:tasks.rados.rados.0.smithi055.stdout:282: done (1 left) 2023-11-28T14:33:08.133 INFO:tasks.rados.rados.0.smithi055.stdout:283: done (0 left) 2023-11-28T14:33:08.133 INFO:tasks.rados.rados.0.smithi055.stdout:284: write oid 38 current snap is 29 2023-11-28T14:33:08.134 INFO:tasks.rados.rados.0.smithi055.stdout:284: seq_num 157 ranges {638699=541153,1770967=531722} 2023-11-28T14:33:08.139 INFO:tasks.rados.rados.0.smithi055.stdout:284: writing smithi05583006-38 from 638699 to 1179852 tid 1 2023-11-28T14:33:08.146 INFO:tasks.rados.rados.0.smithi055.stdout:284: writing smithi05583006-38 from 1770967 to 2302689 tid 2 2023-11-28T14:33:08.146 INFO:tasks.rados.rados.0.smithi055.stdout:285: read oid 40 snap -1 2023-11-28T14:33:08.146 INFO:tasks.rados.rados.0.smithi055.stdout:285: expect (ObjNum 122 snap 18 seq_num 122) 2023-11-28T14:33:08.146 INFO:tasks.rados.rados.0.smithi055.stdout:286: write (excl) oid 15 current snap is 29 2023-11-28T14:33:08.146 INFO:tasks.rados.rados.0.smithi055.stdout:286: seq_num 158 ranges {655506=616402,1829449=675380} 2023-11-28T14:33:08.154 INFO:tasks.rados.rados.0.smithi055.stdout:286: writing smithi05583006-15 from 655506 to 1271908 tid 1 2023-11-28T14:33:08.163 INFO:tasks.rados.rados.0.smithi055.stdout:286: writing smithi05583006-15 from 1829449 to 2504829 tid 2 2023-11-28T14:33:08.163 INFO:tasks.rados.rados.0.smithi055.stdout:287: snap_create 2023-11-28T14:33:08.198 INFO:tasks.rados.rados.0.smithi055.stdout:285: expect (ObjNum 122 snap 18 seq_num 122) 2023-11-28T14:33:08.320 INFO:tasks.rados.rados.0.smithi055.stdout:284: finishing write tid 1 to smithi05583006-38 2023-11-28T14:33:08.320 INFO:tasks.rados.rados.0.smithi055.stdout:286: finishing write tid 1 to smithi05583006-15 2023-11-28T14:33:08.320 INFO:tasks.rados.rados.0.smithi055.stdout:286: finishing write tid 2 to smithi05583006-15 2023-11-28T14:33:08.320 INFO:tasks.rados.rados.0.smithi055.stdout:284: finishing write tid 2 to smithi05583006-38 2023-11-28T14:33:08.320 INFO:tasks.rados.rados.0.smithi055.stdout:284: finishing write tid 3 to smithi05583006-38 2023-11-28T14:33:08.320 INFO:tasks.rados.rados.0.smithi055.stdout:286: finishing write tid 3 to smithi05583006-15 2023-11-28T14:33:08.320 INFO:tasks.rados.rados.0.smithi055.stdout:286: finishing write tid 4 to smithi05583006-15 2023-11-28T14:33:08.320 INFO:tasks.rados.rados.0.smithi055.stdout:286: oid 15 updating version 0 to 38 2023-11-28T14:33:08.321 INFO:tasks.rados.rados.0.smithi055.stdout:286: oid 15 version 38 is already newer than 37 2023-11-28T14:33:08.321 INFO:tasks.rados.rados.0.smithi055.stdout:286: oid 15 version 38 is already newer than 36 2023-11-28T14:33:08.321 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 15 v 38 (ObjNum 158 snap 29 seq_num 158) dirty exists 2023-11-28T14:33:08.321 INFO:tasks.rados.rados.0.smithi055.stdout:286: left oid 15 (ObjNum 158 snap 29 seq_num 158) 2023-11-28T14:33:08.321 INFO:tasks.rados.rados.0.smithi055.stdout:284: finishing write tid 4 to smithi05583006-38 2023-11-28T14:33:08.321 INFO:tasks.rados.rados.0.smithi055.stdout:284: oid 38 updating version 0 to 39 2023-11-28T14:33:08.321 INFO:tasks.rados.rados.0.smithi055.stdout:284: oid 38 version 39 is already newer than 39 2023-11-28T14:33:08.321 INFO:tasks.rados.rados.0.smithi055.stdout:284: oid 38 version 39 is already newer than 35 2023-11-28T14:33:08.321 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 38 v 39 (ObjNum 157 snap 29 seq_num 157) dirty exists 2023-11-28T14:33:08.321 INFO:tasks.rados.rados.0.smithi055.stdout:284: left oid 38 (ObjNum 157 snap 29 seq_num 157) 2023-11-28T14:33:08.896 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:33:09.012 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:09.010+0000 7fc1cd71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:09.015 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:09.014+0000 7fc1cd71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:09.146 INFO:tasks.rados.rados.0.smithi055.stdout:284: done (3 left) 2023-11-28T14:33:09.147 INFO:tasks.rados.rados.0.smithi055.stdout:285: done (2 left) 2023-11-28T14:33:09.147 INFO:tasks.rados.rados.0.smithi055.stdout:286: done (1 left) 2023-11-28T14:33:09.147 INFO:tasks.rados.rados.0.smithi055.stdout:287: done (0 left) 2023-11-28T14:33:09.147 INFO:tasks.rados.rados.0.smithi055.stdout:288: read oid 16 snap -1 2023-11-28T14:33:09.147 INFO:tasks.rados.rados.0.smithi055.stdout:288: expect (ObjNum 72 snap 1 seq_num 72) 2023-11-28T14:33:09.147 INFO:tasks.rados.rados.0.smithi055.stdout:289: write (excl) oid 4 current snap is 30 2023-11-28T14:33:09.147 INFO:tasks.rados.rados.0.smithi055.stdout:289: seq_num 159 ranges {672313=691651,1887931=419038} 2023-11-28T14:33:09.155 INFO:tasks.rados.rados.0.smithi055.stdout:289: writing smithi05583006-4 from 672313 to 1363964 tid 1 2023-11-28T14:33:09.161 INFO:tasks.rados.rados.0.smithi055.stdout:289: writing smithi05583006-4 from 1887931 to 2306969 tid 2 2023-11-28T14:33:09.161 INFO:tasks.rados.rados.0.smithi055.stdout:290: snap_remove snap 27 2023-11-28T14:33:10.151 INFO:tasks.rados.rados.0.smithi055.stdout:290: done (2 left) 2023-11-28T14:33:16.420 INFO:tasks.rados.rados.0.smithi055.stdout:291: snap_create 2023-11-28T14:33:16.421 INFO:tasks.rados.rados.0.smithi055.stdout:288: expect (ObjNum 72 snap 1 seq_num 72) 2023-11-28T14:33:16.421 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:33:16.422 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:33:16.422 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:33:16.422 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-11-28T14:33:16.423 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-11-28T14:33:16.437 INFO:tasks.rados.rados.0.smithi055.stdout:289: finishing write tid 1 to smithi05583006-4 2023-11-28T14:33:16.463 INFO:tasks.rados.rados.0.smithi055.stdout:289: finishing write tid 2 to smithi05583006-4 2023-11-28T14:33:16.463 INFO:tasks.rados.rados.0.smithi055.stdout:289: finishing write tid 3 to smithi05583006-4 2023-11-28T14:33:16.463 INFO:tasks.rados.rados.0.smithi055.stdout:289: finishing write tid 4 to smithi05583006-4 2023-11-28T14:33:16.463 INFO:tasks.rados.rados.0.smithi055.stdout:289: oid 4 updating version 0 to 36 2023-11-28T14:33:16.463 INFO:tasks.rados.rados.0.smithi055.stdout:289: oid 4 version 36 is already newer than 35 2023-11-28T14:33:16.463 INFO:tasks.rados.rados.0.smithi055.stdout:289: oid 4 version 36 is already newer than 34 2023-11-28T14:33:16.464 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 4 v 36 (ObjNum 159 snap 30 seq_num 159) dirty exists 2023-11-28T14:33:16.464 INFO:tasks.rados.rados.0.smithi055.stdout:289: left oid 4 (ObjNum 159 snap 30 seq_num 159) 2023-11-28T14:33:16.464 INFO:tasks.rados.rados.0.smithi055.stdout:288: done (2 left) 2023-11-28T14:33:16.464 INFO:tasks.rados.rados.0.smithi055.stdout:289: done (1 left) 2023-11-28T14:33:16.464 INFO:tasks.rados.rados.0.smithi055.stdout:291: done (0 left) 2023-11-28T14:33:16.464 INFO:tasks.rados.rados.0.smithi055.stdout:292: write oid 32 current snap is 31 2023-11-28T14:33:16.465 INFO:tasks.rados.rados.0.smithi055.stdout:292: seq_num 160 ranges {689120=483253,1946413=679049} 2023-11-28T14:33:16.465 INFO:tasks.rados.rados.0.smithi055.stdout:292: writing smithi05583006-32 from 689120 to 1172373 tid 1 2023-11-28T14:33:16.484 INFO:tasks.rados.rados.0.smithi055.stdout:292: writing smithi05583006-32 from 1946413 to 2625462 tid 2 2023-11-28T14:33:16.484 INFO:tasks.rados.rados.0.smithi055.stdout:293: read oid 36 snap 26 2023-11-28T14:33:16.484 INFO:tasks.rados.rados.0.smithi055.stdout:293: expect deleted 2023-11-28T14:33:16.484 INFO:tasks.rados.rados.0.smithi055.stdout:294: read oid 4 snap -1 2023-11-28T14:33:16.484 INFO:tasks.rados.rados.0.smithi055.stdout:294: expect (ObjNum 159 snap 30 seq_num 159) 2023-11-28T14:33:16.484 INFO:tasks.rados.rados.0.smithi055.stdout:295: snap_remove snap 29 2023-11-28T14:33:16.484 INFO:tasks.rados.rados.0.smithi055.stdout:295: done (3 left) 2023-11-28T14:33:16.484 INFO:tasks.rados.rados.0.smithi055.stdout:296: snap_create 2023-11-28T14:33:16.485 INFO:tasks.rados.rados.0.smithi055.stdout:294: expect (ObjNum 159 snap 30 seq_num 159) 2023-11-28T14:33:16.485 INFO:tasks.rados.rados.0.smithi055.stdout:292: finishing write tid 1 to smithi05583006-32 2023-11-28T14:33:16.485 INFO:tasks.rados.rados.0.smithi055.stdout:292: finishing write tid 2 to smithi05583006-32 2023-11-28T14:33:16.485 INFO:tasks.rados.rados.0.smithi055.stdout:292: finishing write tid 3 to smithi05583006-32 2023-11-28T14:33:16.485 INFO:tasks.rados.rados.0.smithi055.stdout:292: finishing write tid 4 to smithi05583006-32 2023-11-28T14:33:16.485 INFO:tasks.rados.rados.0.smithi055.stdout:292: oid 32 updating version 0 to 108 2023-11-28T14:33:16.485 INFO:tasks.rados.rados.0.smithi055.stdout:292: oid 32 updating version 108 to 109 2023-11-28T14:33:16.485 INFO:tasks.rados.rados.0.smithi055.stdout:292: oid 32 version 109 is already newer than 107 2023-11-28T14:33:16.485 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 32 v 109 (ObjNum 160 snap 31 seq_num 160) dirty exists 2023-11-28T14:33:16.485 INFO:tasks.rados.rados.0.smithi055.stdout:292: left oid 32 (ObjNum 160 snap 31 seq_num 160) 2023-11-28T14:33:16.485 INFO:tasks.rados.rados.0.smithi055.stdout:292: done (3 left) 2023-11-28T14:33:16.485 INFO:tasks.rados.rados.0.smithi055.stdout:293: done (2 left) 2023-11-28T14:33:16.485 INFO:tasks.rados.rados.0.smithi055.stdout:294: done (1 left) 2023-11-28T14:33:16.486 INFO:tasks.rados.rados.0.smithi055.stdout:296: done (0 left) 2023-11-28T14:33:16.486 INFO:tasks.rados.rados.0.smithi055.stdout:297: read oid 34 snap -1 2023-11-28T14:33:16.486 INFO:tasks.rados.rados.0.smithi055.stdout:297: expect deleted 2023-11-28T14:33:16.486 INFO:tasks.rados.rados.0.smithi055.stdout:298: write oid 28 current snap is 32 2023-11-28T14:33:16.486 INFO:tasks.rados.rados.0.smithi055.stdout:298: seq_num 161 ranges {705927=558502,2004895=422707} 2023-11-28T14:33:16.486 INFO:tasks.rados.rados.0.smithi055.stdout:298: writing smithi05583006-28 from 705927 to 1264429 tid 1 2023-11-28T14:33:16.486 INFO:tasks.rados.rados.0.smithi055.stdout:298: writing smithi05583006-28 from 2004895 to 2427602 tid 2 2023-11-28T14:33:16.486 INFO:tasks.rados.rados.0.smithi055.stdout:299: snap_create 2023-11-28T14:33:16.486 INFO:tasks.rados.rados.0.smithi055.stdout:298: finishing write tid 1 to smithi05583006-28 2023-11-28T14:33:16.486 INFO:tasks.rados.rados.0.smithi055.stdout:298: finishing write tid 2 to smithi05583006-28 2023-11-28T14:33:16.486 INFO:tasks.rados.rados.0.smithi055.stdout:298: finishing write tid 3 to smithi05583006-28 2023-11-28T14:33:16.486 INFO:tasks.rados.rados.0.smithi055.stdout:298: finishing write tid 4 to smithi05583006-28 2023-11-28T14:33:16.486 INFO:tasks.rados.rados.0.smithi055.stdout:298: oid 28 updating version 0 to 45 2023-11-28T14:33:16.487 INFO:tasks.rados.rados.0.smithi055.stdout:298: oid 28 version 45 is already newer than 45 2023-11-28T14:33:16.487 INFO:tasks.rados.rados.0.smithi055.stdout:298: oid 28 updating version 45 to 46 2023-11-28T14:33:16.487 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 28 v 46 (ObjNum 161 snap 32 seq_num 161) dirty exists 2023-11-28T14:33:16.487 INFO:tasks.rados.rados.0.smithi055.stdout:298: left oid 28 (ObjNum 161 snap 32 seq_num 161) 2023-11-28T14:33:16.487 INFO:tasks.rados.rados.0.smithi055.stdout:297: done (2 left) 2023-11-28T14:33:16.487 INFO:tasks.rados.rados.0.smithi055.stdout:298: done (1 left) 2023-11-28T14:33:16.487 INFO:tasks.rados.rados.0.smithi055.stdout:299: done (0 left) 2023-11-28T14:33:16.487 INFO:tasks.rados.rados.0.smithi055.stdout:300: read oid 2 snap -1 2023-11-28T14:33:16.487 INFO:tasks.rados.rados.0.smithi055.stdout:300: expect (ObjNum 155 snap 28 seq_num 155) 2023-11-28T14:33:16.487 INFO:tasks.rados.rados.0.smithi055.stdout:301: write oid 38 current snap is 33 2023-11-28T14:33:16.487 INFO:tasks.rados.rados.0.smithi055.stdout:301: seq_num 162 ranges {722734=633751,2063377=659357} 2023-11-28T14:33:16.487 INFO:tasks.rados.rados.0.smithi055.stdout:301: writing smithi05583006-38 from 722734 to 1356485 tid 1 2023-11-28T14:33:16.487 INFO:tasks.rados.rados.0.smithi055.stdout:301: writing smithi05583006-38 from 2063377 to 2722734 tid 2 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:302: delete oid 13 current snap is 33 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:300: expect (ObjNum 155 snap 28 seq_num 155) 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:301: finishing write tid 1 to smithi05583006-38 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:301: finishing write tid 2 to smithi05583006-38 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:301: finishing write tid 3 to smithi05583006-38 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:301: finishing write tid 4 to smithi05583006-38 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:301: oid 38 updating version 0 to 44 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:301: oid 38 updating version 44 to 45 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:301: oid 38 version 45 is already newer than 43 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 38 v 45 (ObjNum 162 snap 33 seq_num 162) dirty exists 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:301: left oid 38 (ObjNum 162 snap 33 seq_num 162) 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:300: done (2 left) 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:301: done (1 left) 2023-11-28T14:33:16.488 INFO:tasks.rados.rados.0.smithi055.stdout:302: done (0 left) 2023-11-28T14:33:16.489 INFO:tasks.rados.rados.0.smithi055.stdout:303: snap_create 2023-11-28T14:33:16.489 INFO:tasks.rados.rados.0.smithi055.stdout:303: done (0 left) 2023-11-28T14:33:16.489 INFO:tasks.rados.rados.0.smithi055.stdout:304: delete oid 35 current snap is 34 2023-11-28T14:33:16.489 INFO:tasks.rados.rados.0.smithi055.stdout:304: done (0 left) 2023-11-28T14:33:16.489 INFO:tasks.rados.rados.0.smithi055.stdout:305: delete oid 46 current snap is 34 2023-11-28T14:33:16.489 INFO:tasks.rados.rados.0.smithi055.stdout:305: done (0 left) 2023-11-28T14:33:16.489 INFO:tasks.rados.rados.0.smithi055.stdout:306: write oid 26 current snap is 34 2023-11-28T14:33:16.489 INFO:tasks.rados.rados.0.smithi055.stdout:306: seq_num 166 ranges {789962=534747,1780952=690056} 2023-11-28T14:33:16.489 INFO:tasks.rados.rados.0.smithi055.stdout:306: writing smithi05583006-26 from 789962 to 1324709 tid 1 2023-11-28T14:33:16.489 INFO:tasks.rados.rados.0.smithi055.stdout:306: writing smithi05583006-26 from 1780952 to 2471008 tid 2 2023-11-28T14:33:16.489 INFO:tasks.rados.rados.0.smithi055.stdout:307: snap_remove snap 26 2023-11-28T14:33:16.489 INFO:tasks.rados.rados.0.smithi055.stdout:307: done (1 left) 2023-11-28T14:33:16.490 INFO:tasks.rados.rados.0.smithi055.stdout:308: snap_create 2023-11-28T14:33:16.490 INFO:tasks.rados.rados.0.smithi055.stdout:306: finishing write tid 1 to smithi05583006-26 2023-11-28T14:33:16.490 INFO:tasks.rados.rados.0.smithi055.stdout:306: finishing write tid 2 to smithi05583006-26 2023-11-28T14:33:16.490 INFO:tasks.rados.rados.0.smithi055.stdout:306: finishing write tid 3 to smithi05583006-26 2023-11-28T14:33:16.490 INFO:tasks.rados.rados.0.smithi055.stdout:306: finishing write tid 4 to smithi05583006-26 2023-11-28T14:33:16.490 INFO:tasks.rados.rados.0.smithi055.stdout:306: oid 26 updating version 0 to 36 2023-11-28T14:33:16.490 INFO:tasks.rados.rados.0.smithi055.stdout:306: oid 26 version 36 is already newer than 35 2023-11-28T14:33:16.490 INFO:tasks.rados.rados.0.smithi055.stdout:306: oid 26 version 36 is already newer than 34 2023-11-28T14:33:16.490 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 26 v 36 (ObjNum 166 snap 34 seq_num 166) dirty exists 2023-11-28T14:33:16.490 INFO:tasks.rados.rados.0.smithi055.stdout:306: left oid 26 (ObjNum 166 snap 34 seq_num 166) 2023-11-28T14:33:16.617 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:16.615+0000 7fa1ff91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:16.621 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:16.615+0000 7f6baaf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:16.621 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:16.620+0000 7fa1ff91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:16.624 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:16.623+0000 7f6baaf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:16.637 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:33:16.638 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:33:16.638 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:33:16.638 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:33:16.638 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:33:16.638 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:33:16.638 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:33:16.638 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:33:16.638 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:33:16.638 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:33:16.638 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:33:16.639 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:33:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:33:16.641 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:33:16.641 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-11-28T14:33:16.755 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:16.750+0000 7fe2af46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:16.758 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:16.757+0000 7fe2af46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:16.771 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:33:16.771 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:33:16.771 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:33:16.771 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:33:16.771 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:33:16.771 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:33:16.771 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:33:16.771 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:33:16.771 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:33:16.771 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:33:16.772 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:33:16.772 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:33:16.774 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:33:16.774 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:33:16.774 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-11-28T14:33:16.886 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:16.881+0000 7fab4651e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:16.889 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:16.888+0000 7fab4651e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:16.903 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:33:16.903 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:33:16.903 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:33:16.903 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:33:16.903 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:33:16.903 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:33:16.903 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:33:16.904 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:33:16.904 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:33:16.904 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:33:16.904 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:33:16.904 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:33:16.905 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:33:16.906 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:33:16.906 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-11-28T14:33:17.019 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:17.014+0000 7f88bc11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:17.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:17.021+0000 7f88bc11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:17.036 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:33:17.036 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:33:17.036 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:33:17.036 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:33:17.036 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:33:17.036 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:33:17.036 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:33:17.036 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:33:17.037 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:33:17.037 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:33:17.037 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:33:17.037 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:33:17.038 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:33:17.039 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:33:17.207 INFO:tasks.rados.rados.0.smithi055.stdout:306: done (1 left) 2023-11-28T14:33:17.207 INFO:tasks.rados.rados.0.smithi055.stdout:308: done (0 left) 2023-11-28T14:33:17.207 INFO:tasks.rados.rados.0.smithi055.stdout:309: write (excl) oid 29 current snap is 35 2023-11-28T14:33:17.207 INFO:tasks.rados.rados.0.smithi055.stdout:309: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2023-11-28T14:33:17.213 INFO:tasks.rados.rados.0.smithi055.stdout:309: writing smithi05583006-29 from 406769 to 1016765 tid 1 2023-11-28T14:33:17.219 INFO:tasks.rados.rados.0.smithi055.stdout:309: writing smithi05583006-29 from 1439434 to 1873148 tid 2 2023-11-28T14:33:17.225 INFO:tasks.rados.rados.0.smithi055.stdout:309: writing smithi05583006-29 from 2303522 to 2806769 tid 3 2023-11-28T14:33:17.225 INFO:tasks.rados.rados.0.smithi055.stdout:310: snap_remove snap 30 2023-11-28T14:33:18.212 INFO:tasks.rados.rados.0.smithi055.stdout:310: done (1 left) 2023-11-28T14:33:18.212 INFO:tasks.rados.rados.0.smithi055.stdout:311: write (excl) oid 28 current snap is 35 2023-11-28T14:33:18.212 INFO:tasks.rados.rados.0.smithi055.stdout:309: finishing write tid 1 to smithi05583006-29 2023-11-28T14:33:18.213 INFO:tasks.rados.rados.0.smithi055.stdout:311: seq_num 168 ranges {423576=401598,1497916=693725} 2023-11-28T14:33:18.216 INFO:tasks.rados.rados.0.smithi055.stdout:311: writing smithi05583006-28 from 423576 to 825174 tid 1 2023-11-28T14:33:18.225 INFO:tasks.rados.rados.0.smithi055.stdout:311: writing smithi05583006-28 from 1497916 to 2191641 tid 2 2023-11-28T14:33:18.225 INFO:tasks.rados.rados.0.smithi055.stdout:312: write oid 19 current snap is 35 2023-11-28T14:33:18.225 INFO:tasks.rados.rados.0.smithi055.stdout:309: finishing write tid 2 to smithi05583006-29 2023-11-28T14:33:18.225 INFO:tasks.rados.rados.0.smithi055.stdout:309: finishing write tid 3 to smithi05583006-29 2023-11-28T14:33:18.225 INFO:tasks.rados.rados.0.smithi055.stdout:309: finishing write tid 4 to smithi05583006-29 2023-11-28T14:33:18.225 INFO:tasks.rados.rados.0.smithi055.stdout:312: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2023-11-28T14:33:18.231 INFO:tasks.rados.rados.0.smithi055.stdout:312: writing smithi05583006-19 from 440383 to 917230 tid 1 2023-11-28T14:33:18.237 INFO:tasks.rados.rados.0.smithi055.stdout:312: writing smithi05583006-19 from 1556398 to 1993781 tid 2 2023-11-28T14:33:18.243 INFO:tasks.rados.rados.0.smithi055.stdout:312: writing smithi05583006-19 from 2474721 to 2840383 tid 3 2023-11-28T14:33:18.243 INFO:tasks.rados.rados.0.smithi055.stdout:313: read oid 41 snap 31 2023-11-28T14:33:18.243 INFO:tasks.rados.rados.0.smithi055.stdout:313: expect (ObjNum 107 snap 13 seq_num 107) 2023-11-28T14:33:18.243 INFO:tasks.rados.rados.0.smithi055.stdout:314: snap_create 2023-11-28T14:33:18.243 INFO:tasks.rados.rados.0.smithi055.stdout:309: finishing write tid 5 to smithi05583006-29 2023-11-28T14:33:18.244 INFO:tasks.rados.rados.0.smithi055.stdout:309: oid 29 updating version 0 to 46 2023-11-28T14:33:18.244 INFO:tasks.rados.rados.0.smithi055.stdout:309: oid 29 version 46 is already newer than 41 2023-11-28T14:33:18.244 INFO:tasks.rados.rados.0.smithi055.stdout:309: oid 29 version 46 is already newer than 45 2023-11-28T14:33:18.244 INFO:tasks.rados.rados.0.smithi055.stdout:309: oid 29 version 46 is already newer than 42 2023-11-28T14:33:18.244 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 29 v 46 (ObjNum 167 snap 35 seq_num 167) dirty exists 2023-11-28T14:33:18.244 INFO:tasks.rados.rados.0.smithi055.stdout:309: left oid 29 (ObjNum 167 snap 35 seq_num 167) 2023-11-28T14:33:18.314 INFO:tasks.rados.rados.0.smithi055.stdout:311: finishing write tid 1 to smithi05583006-28 2023-11-28T14:33:18.324 INFO:tasks.rados.rados.0.smithi055.stdout:312: finishing write tid 1 to smithi05583006-19 2023-11-28T14:33:18.338 INFO:tasks.rados.rados.0.smithi055.stdout:313: expect (ObjNum 107 snap 13 seq_num 107) 2023-11-28T14:33:18.589 INFO:tasks.rados.rados.0.smithi055.stdout:311: finishing write tid 2 to smithi05583006-28 2023-11-28T14:33:18.589 INFO:tasks.rados.rados.0.smithi055.stdout:311: finishing write tid 3 to smithi05583006-28 2023-11-28T14:33:18.589 INFO:tasks.rados.rados.0.smithi055.stdout:311: finishing write tid 4 to smithi05583006-28 2023-11-28T14:33:18.589 INFO:tasks.rados.rados.0.smithi055.stdout:311: oid 28 updating version 0 to 48 2023-11-28T14:33:18.589 INFO:tasks.rados.rados.0.smithi055.stdout:311: oid 28 updating version 48 to 50 2023-11-28T14:33:18.589 INFO:tasks.rados.rados.0.smithi055.stdout:311: oid 28 version 50 is already newer than 49 2023-11-28T14:33:18.589 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 28 v 50 (ObjNum 168 snap 35 seq_num 168) dirty exists 2023-11-28T14:33:18.589 INFO:tasks.rados.rados.0.smithi055.stdout:311: left oid 28 (ObjNum 168 snap 35 seq_num 168) 2023-11-28T14:33:18.589 INFO:tasks.rados.rados.0.smithi055.stdout:312: finishing write tid 2 to smithi05583006-19 2023-11-28T14:33:18.589 INFO:tasks.rados.rados.0.smithi055.stdout:312: finishing write tid 3 to smithi05583006-19 2023-11-28T14:33:18.590 INFO:tasks.rados.rados.0.smithi055.stdout:312: finishing write tid 4 to smithi05583006-19 2023-11-28T14:33:18.590 INFO:tasks.rados.rados.0.smithi055.stdout:312: finishing write tid 5 to smithi05583006-19 2023-11-28T14:33:18.590 INFO:tasks.rados.rados.0.smithi055.stdout:312: oid 19 updating version 0 to 43 2023-11-28T14:33:18.590 INFO:tasks.rados.rados.0.smithi055.stdout:312: oid 19 version 43 is already newer than 42 2023-11-28T14:33:18.590 INFO:tasks.rados.rados.0.smithi055.stdout:312: oid 19 updating version 43 to 45 2023-11-28T14:33:18.590 INFO:tasks.rados.rados.0.smithi055.stdout:312: oid 19 version 45 is already newer than 44 2023-11-28T14:33:18.590 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 19 v 45 (ObjNum 169 snap 35 seq_num 169) dirty exists 2023-11-28T14:33:18.590 INFO:tasks.rados.rados.0.smithi055.stdout:312: left oid 19 (ObjNum 169 snap 35 seq_num 169) 2023-11-28T14:33:18.695 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:33:18.806 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:18.803+0000 7f2ee411e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:18.809 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:18.808+0000 7f2ee411e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:19.219 INFO:tasks.rados.rados.0.smithi055.stdout:309: done (4 left) 2023-11-28T14:33:19.219 INFO:tasks.rados.rados.0.smithi055.stdout:311: done (3 left) 2023-11-28T14:33:19.219 INFO:tasks.rados.rados.0.smithi055.stdout:312: done (2 left) 2023-11-28T14:33:19.219 INFO:tasks.rados.rados.0.smithi055.stdout:313: done (1 left) 2023-11-28T14:33:19.219 INFO:tasks.rados.rados.0.smithi055.stdout:314: done (0 left) 2023-11-28T14:33:19.219 INFO:tasks.rados.rados.0.smithi055.stdout:315: write oid 27 current snap is 36 2023-11-28T14:33:19.219 INFO:tasks.rados.rados.0.smithi055.stdout:315: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2023-11-28T14:33:19.224 INFO:tasks.rados.rados.0.smithi055.stdout:315: writing smithi05583006-27 from 457190 to 1009286 tid 1 2023-11-28T14:33:19.230 INFO:tasks.rados.rados.0.smithi055.stdout:315: writing smithi05583006-27 from 1614880 to 2195921 tid 2 2023-11-28T14:33:19.234 INFO:tasks.rados.rados.0.smithi055.stdout:315: writing smithi05583006-27 from 2785791 to 2857190 tid 3 2023-11-28T14:33:19.234 INFO:tasks.rados.rados.0.smithi055.stdout:316: read oid 43 snap -1 2023-11-28T14:33:19.234 INFO:tasks.rados.rados.0.smithi055.stdout:316: expect deleted 2023-11-28T14:33:19.234 INFO:tasks.rados.rados.0.smithi055.stdout:317: write oid 41 current snap is 36 2023-11-28T14:33:19.235 INFO:tasks.rados.rados.0.smithi055.stdout:317: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2023-11-28T14:33:19.240 INFO:tasks.rados.rados.0.smithi055.stdout:317: writing smithi05583006-41 from 473997 to 1101342 tid 1 2023-11-28T14:33:19.247 INFO:tasks.rados.rados.0.smithi055.stdout:317: writing smithi05583006-41 from 1673362 to 2114414 tid 2 2023-11-28T14:33:19.253 INFO:tasks.rados.rados.0.smithi055.stdout:317: writing smithi05583006-41 from 2529567 to 2873997 tid 3 2023-11-28T14:33:19.253 INFO:tasks.rados.rados.0.smithi055.stdout:318: snap_remove snap 33 2023-11-28T14:33:20.220 INFO:tasks.rados.rados.0.smithi055.stdout:318: done (3 left) 2023-11-28T14:33:20.221 INFO:tasks.rados.rados.0.smithi055.stdout:319: read oid 11 snap -1 2023-11-28T14:33:20.221 INFO:tasks.rados.rados.0.smithi055.stdout:319: expect deleted 2023-11-28T14:33:20.221 INFO:tasks.rados.rados.0.smithi055.stdout:320: snap_create 2023-11-28T14:33:20.221 INFO:tasks.rados.rados.0.smithi055.stdout:317: finishing write tid 1 to smithi05583006-41 2023-11-28T14:33:20.221 INFO:tasks.rados.rados.0.smithi055.stdout:315: finishing write tid 1 to smithi05583006-27 2023-11-28T14:33:20.221 INFO:tasks.rados.rados.0.smithi055.stdout:315: finishing write tid 2 to smithi05583006-27 2023-11-28T14:33:20.221 INFO:tasks.rados.rados.0.smithi055.stdout:315: finishing write tid 3 to smithi05583006-27 2023-11-28T14:33:20.221 INFO:tasks.rados.rados.0.smithi055.stdout:317: finishing write tid 2 to smithi05583006-41 2023-11-28T14:33:20.221 INFO:tasks.rados.rados.0.smithi055.stdout:317: finishing write tid 3 to smithi05583006-41 2023-11-28T14:33:20.221 INFO:tasks.rados.rados.0.smithi055.stdout:315: finishing write tid 4 to smithi05583006-27 2023-11-28T14:33:20.222 INFO:tasks.rados.rados.0.smithi055.stdout:315: finishing write tid 5 to smithi05583006-27 2023-11-28T14:33:20.222 INFO:tasks.rados.rados.0.smithi055.stdout:315: oid 27 updating version 0 to 35 2023-11-28T14:33:20.222 INFO:tasks.rados.rados.0.smithi055.stdout:315: oid 27 updating version 35 to 36 2023-11-28T14:33:20.222 INFO:tasks.rados.rados.0.smithi055.stdout:315: oid 27 version 36 is already newer than 35 2023-11-28T14:33:20.222 INFO:tasks.rados.rados.0.smithi055.stdout:315: oid 27 version 36 is already newer than 33 2023-11-28T14:33:20.222 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 27 v 36 (ObjNum 170 snap 36 seq_num 170) dirty exists 2023-11-28T14:33:20.222 INFO:tasks.rados.rados.0.smithi055.stdout:315: left oid 27 (ObjNum 170 snap 36 seq_num 170) 2023-11-28T14:33:20.222 INFO:tasks.rados.rados.0.smithi055.stdout:317: finishing write tid 4 to smithi05583006-41 2023-11-28T14:33:20.222 INFO:tasks.rados.rados.0.smithi055.stdout:317: finishing write tid 5 to smithi05583006-41 2023-11-28T14:33:20.222 INFO:tasks.rados.rados.0.smithi055.stdout:317: oid 41 updating version 0 to 142 2023-11-28T14:33:20.222 INFO:tasks.rados.rados.0.smithi055.stdout:317: oid 41 version 142 is already newer than 140 2023-11-28T14:33:20.223 INFO:tasks.rados.rados.0.smithi055.stdout:317: oid 41 version 142 is already newer than 142 2023-11-28T14:33:20.223 INFO:tasks.rados.rados.0.smithi055.stdout:317: oid 41 updating version 142 to 143 2023-11-28T14:33:20.223 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 41 v 143 (ObjNum 171 snap 36 seq_num 171) dirty exists 2023-11-28T14:33:20.223 INFO:tasks.rados.rados.0.smithi055.stdout:317: left oid 41 (ObjNum 171 snap 36 seq_num 171) 2023-11-28T14:33:20.881 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:33:21.004 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:21.002+0000 7fbbbdf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:21.008 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:21.007+0000 7fbbbdf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:21.239 INFO:tasks.rados.rados.0.smithi055.stdout:315: done (4 left) 2023-11-28T14:33:21.239 INFO:tasks.rados.rados.0.smithi055.stdout:316: done (3 left) 2023-11-28T14:33:21.239 INFO:tasks.rados.rados.0.smithi055.stdout:317: done (2 left) 2023-11-28T14:33:21.240 INFO:tasks.rados.rados.0.smithi055.stdout:319: done (1 left) 2023-11-28T14:33:21.240 INFO:tasks.rados.rados.0.smithi055.stdout:320: done (0 left) 2023-11-28T14:33:21.240 INFO:tasks.rados.rados.0.smithi055.stdout:321: read oid 24 snap 28 2023-11-28T14:33:21.240 INFO:tasks.rados.rados.0.smithi055.stdout:321: expect (ObjNum 134 snap 18 seq_num 134) 2023-11-28T14:33:21.240 INFO:tasks.rados.rados.0.smithi055.stdout:322: snap_create 2023-11-28T14:33:21.300 INFO:tasks.rados.rados.0.smithi055.stdout:321: expect (ObjNum 134 snap 18 seq_num 134) 2023-11-28T14:33:22.039 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:33:22.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:33:22.039 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-11-28T14:33:22.040 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-11-28T14:33:22.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:22.154+0000 7fcf042f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:22.163 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:22.161+0000 7fcf042f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:22.176 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:33:22.176 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:33:22.176 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:33:22.176 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:33:22.176 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:33:22.176 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:33:22.177 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:33:22.177 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:33:22.177 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:33:22.177 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:33:22.177 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:33:22.177 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:33:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:33:22.179 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:33:22.179 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-11-28T14:33:22.245 INFO:tasks.rados.rados.0.smithi055.stdout:321: done (1 left) 2023-11-28T14:33:22.245 INFO:tasks.rados.rados.0.smithi055.stdout:322: done (0 left) 2023-11-28T14:33:22.245 INFO:tasks.rados.rados.0.smithi055.stdout:323: delete oid 33 current snap is 38 2023-11-28T14:33:22.298 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:22.292+0000 7fab5602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:22.301 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:22.299+0000 7fab5602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:22.310 INFO:tasks.rados.rados.0.smithi055.stdout:323: done (0 left) 2023-11-28T14:33:22.310 INFO:tasks.rados.rados.0.smithi055.stdout:324: snap_create 2023-11-28T14:33:22.314 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:33:22.314 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:33:22.314 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:33:22.314 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:33:22.314 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:33:22.314 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:33:22.314 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:33:22.314 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:33:22.315 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:33:22.315 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:33:22.315 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:33:22.315 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:33:22.317 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:33:22.317 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:33:22.317 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-11-28T14:33:22.430 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:22.424+0000 7f8cbae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:22.433 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:22.431+0000 7f8cbae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:22.448 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:33:22.448 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:33:22.448 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:33:22.448 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:33:22.448 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:33:22.448 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:33:22.448 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:33:22.448 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:33:22.449 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:33:22.449 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:33:22.449 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:33:22.449 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:33:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:33:22.451 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:33:22.451 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-11-28T14:33:22.565 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:22.559+0000 7ff0ced5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:22.568 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:22.566+0000 7ff0ced5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:22.581 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:33:22.582 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:33:22.582 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:33:22.582 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:33:22.582 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:33:22.582 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:33:22.582 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:33:22.582 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:33:22.582 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:33:22.582 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:33:22.582 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:33:22.583 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:33:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:33:22.585 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:33:23.080 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:33:23.195 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:23.192+0000 7f74c4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:23.198 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:23.196+0000 7f74c4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:23.256 INFO:tasks.rados.rados.0.smithi055.stdout:324: done (0 left) 2023-11-28T14:33:23.257 INFO:tasks.rados.rados.0.smithi055.stdout:325: write (excl) oid 13 current snap is 39 2023-11-28T14:33:23.257 INFO:tasks.rados.rados.0.smithi055.stdout:325: seq_num 173 ranges {507611=777843,2073973=444721} 2023-11-28T14:33:23.266 INFO:tasks.rados.rados.0.smithi055.stdout:325: writing smithi05583006-13 from 507611 to 1285454 tid 1 2023-11-28T14:33:23.273 INFO:tasks.rados.rados.0.smithi055.stdout:325: writing smithi05583006-13 from 2073973 to 2518694 tid 2 2023-11-28T14:33:23.273 INFO:tasks.rados.rados.0.smithi055.stdout:326: snap_create 2023-11-28T14:33:23.305 INFO:tasks.rados.rados.0.smithi055.stdout:325: finishing write tid 1 to smithi05583006-13 2023-11-28T14:33:23.323 INFO:tasks.rados.rados.0.smithi055.stdout:325: finishing write tid 2 to smithi05583006-13 2023-11-28T14:33:23.328 INFO:tasks.rados.rados.0.smithi055.stdout:325: finishing write tid 3 to smithi05583006-13 2023-11-28T14:33:23.328 INFO:tasks.rados.rados.0.smithi055.stdout:325: finishing write tid 4 to smithi05583006-13 2023-11-28T14:33:23.328 INFO:tasks.rados.rados.0.smithi055.stdout:325: oid 13 updating version 0 to 122 2023-11-28T14:33:23.328 INFO:tasks.rados.rados.0.smithi055.stdout:325: oid 13 updating version 122 to 124 2023-11-28T14:33:23.328 INFO:tasks.rados.rados.0.smithi055.stdout:325: oid 13 version 124 is already newer than 124 2023-11-28T14:33:23.328 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 13 v 124 (ObjNum 173 snap 39 seq_num 173) dirty exists 2023-11-28T14:33:23.328 INFO:tasks.rados.rados.0.smithi055.stdout:325: left oid 13 (ObjNum 173 snap 39 seq_num 173) 2023-11-28T14:33:24.258 INFO:tasks.rados.rados.0.smithi055.stdout:325: done (1 left) 2023-11-28T14:33:24.259 INFO:tasks.rados.rados.0.smithi055.stdout:326: done (0 left) 2023-11-28T14:33:24.259 INFO:tasks.rados.rados.0.smithi055.stdout:327: read oid 49 snap -1 2023-11-28T14:33:24.259 INFO:tasks.rados.rados.0.smithi055.stdout:327: expect (ObjNum 119 snap 18 seq_num 119) 2023-11-28T14:33:24.259 INFO:tasks.rados.rados.0.smithi055.stdout:328: write oid 33 current snap is 40 2023-11-28T14:33:24.259 INFO:tasks.rados.rados.0.smithi055.stdout:328: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2023-11-28T14:33:24.264 INFO:tasks.rados.rados.0.smithi055.stdout:328: writing smithi05583006-33 from 524418 to 977510 tid 1 2023-11-28T14:33:24.271 INFO:tasks.rados.rados.0.smithi055.stdout:328: writing smithi05583006-33 from 1732455 to 2320834 tid 2 2023-11-28T14:33:24.275 INFO:tasks.rados.rados.0.smithi055.stdout:328: writing smithi05583006-33 from 2779130 to 2924418 tid 3 2023-11-28T14:33:24.275 INFO:tasks.rados.rados.0.smithi055.stdout:329: delete oid 14 current snap is 40 2023-11-28T14:33:24.317 INFO:tasks.rados.rados.0.smithi055.stdout:327: expect (ObjNum 119 snap 18 seq_num 119) 2023-11-28T14:33:24.414 INFO:tasks.rados.rados.0.smithi055.stdout:328: finishing write tid 1 to smithi05583006-33 2023-11-28T14:33:24.414 INFO:tasks.rados.rados.0.smithi055.stdout:328: finishing write tid 2 to smithi05583006-33 2023-11-28T14:33:24.414 INFO:tasks.rados.rados.0.smithi055.stdout:328: finishing write tid 3 to smithi05583006-33 2023-11-28T14:33:24.414 INFO:tasks.rados.rados.0.smithi055.stdout:328: finishing write tid 4 to smithi05583006-33 2023-11-28T14:33:24.414 INFO:tasks.rados.rados.0.smithi055.stdout:328: finishing write tid 5 to smithi05583006-33 2023-11-28T14:33:24.414 INFO:tasks.rados.rados.0.smithi055.stdout:328: oid 33 updating version 0 to 125 2023-11-28T14:33:24.414 INFO:tasks.rados.rados.0.smithi055.stdout:328: oid 33 updating version 125 to 128 2023-11-28T14:33:24.415 INFO:tasks.rados.rados.0.smithi055.stdout:328: oid 33 version 128 is already newer than 128 2023-11-28T14:33:24.415 INFO:tasks.rados.rados.0.smithi055.stdout:328: oid 33 version 128 is already newer than 126 2023-11-28T14:33:24.415 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 33 v 128 (ObjNum 174 snap 40 seq_num 174) dirty exists 2023-11-28T14:33:24.415 INFO:tasks.rados.rados.0.smithi055.stdout:328: left oid 33 (ObjNum 174 snap 40 seq_num 174) 2023-11-28T14:33:24.415 INFO:tasks.rados.rados.0.smithi055.stdout:327: done (2 left) 2023-11-28T14:33:24.415 INFO:tasks.rados.rados.0.smithi055.stdout:328: done (1 left) 2023-11-28T14:33:24.415 INFO:tasks.rados.rados.0.smithi055.stdout:329: done (0 left) 2023-11-28T14:33:24.415 INFO:tasks.rados.rados.0.smithi055.stdout:330: read oid 34 snap -1 2023-11-28T14:33:24.415 INFO:tasks.rados.rados.0.smithi055.stdout:330: expect deleted 2023-11-28T14:33:24.415 INFO:tasks.rados.rados.0.smithi055.stdout:331: snap_remove snap 28 2023-11-28T14:33:25.127 INFO:tasks.rados.rados.0.smithi055.stdout:331: done (1 left) 2023-11-28T14:33:25.128 INFO:tasks.rados.rados.0.smithi055.stdout:332: snap_remove snap 39 2023-11-28T14:33:25.271 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:33:25.390 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:25.387+0000 7f1b932de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:25.393 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:25.391+0000 7f1b932de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:26.131 INFO:tasks.rados.rados.0.smithi055.stdout:332: done (1 left) 2023-11-28T14:33:26.131 INFO:tasks.rados.rados.0.smithi055.stdout:333: snap_remove snap 32 2023-11-28T14:33:27.133 INFO:tasks.rados.rados.0.smithi055.stdout:333: done (1 left) 2023-11-28T14:33:27.133 INFO:tasks.rados.rados.0.smithi055.stdout:334: delete oid 20 current snap is 40 2023-11-28T14:33:27.184 INFO:tasks.rados.rados.0.smithi055.stdout:330: done (1 left) 2023-11-28T14:33:27.184 INFO:tasks.rados.rados.0.smithi055.stdout:334: done (0 left) 2023-11-28T14:33:27.184 INFO:tasks.rados.rados.0.smithi055.stdout:335: write (excl) oid 24 current snap is 40 2023-11-28T14:33:27.184 INFO:tasks.rados.rados.0.smithi055.stdout:335: seq_num 177 ranges {574839=795192,1907901=452059} 2023-11-28T14:33:27.191 INFO:tasks.rados.rados.0.smithi055.stdout:335: writing smithi05583006-24 from 574839 to 1370031 tid 1 2023-11-28T14:33:27.197 INFO:tasks.rados.rados.0.smithi055.stdout:335: writing smithi05583006-24 from 1907901 to 2359960 tid 2 2023-11-28T14:33:27.197 INFO:tasks.rados.rados.0.smithi055.stdout:336: read oid 7 snap -1 2023-11-28T14:33:27.197 INFO:tasks.rados.rados.0.smithi055.stdout:336: expect (ObjNum 150 snap 26 seq_num 150) 2023-11-28T14:33:27.197 INFO:tasks.rados.rados.0.smithi055.stdout:337: delete oid 46 current snap is 40 2023-11-28T14:33:27.251 INFO:tasks.rados.rados.0.smithi055.stdout:337: done (2 left) 2023-11-28T14:33:27.251 INFO:tasks.rados.rados.0.smithi055.stdout:338: snap_create 2023-11-28T14:33:27.265 INFO:tasks.rados.rados.0.smithi055.stdout:335: finishing write tid 1 to smithi05583006-24 2023-11-28T14:33:27.273 INFO:tasks.rados.rados.0.smithi055.stdout:336: expect (ObjNum 150 snap 26 seq_num 150) 2023-11-28T14:33:27.428 INFO:tasks.rados.rados.0.smithi055.stdout:335: finishing write tid 2 to smithi05583006-24 2023-11-28T14:33:27.429 INFO:tasks.rados.rados.0.smithi055.stdout:335: finishing write tid 3 to smithi05583006-24 2023-11-28T14:33:27.429 INFO:tasks.rados.rados.0.smithi055.stdout:335: finishing write tid 4 to smithi05583006-24 2023-11-28T14:33:27.429 INFO:tasks.rados.rados.0.smithi055.stdout:335: oid 24 updating version 0 to 31 2023-11-28T14:33:27.429 INFO:tasks.rados.rados.0.smithi055.stdout:335: oid 24 updating version 31 to 32 2023-11-28T14:33:27.429 INFO:tasks.rados.rados.0.smithi055.stdout:335: oid 24 updating version 32 to 33 2023-11-28T14:33:27.429 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 24 v 33 (ObjNum 177 snap 40 seq_num 177) dirty exists 2023-11-28T14:33:27.429 INFO:tasks.rados.rados.0.smithi055.stdout:335: left oid 24 (ObjNum 177 snap 40 seq_num 177) 2023-11-28T14:33:27.464 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:33:27.575 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:27.572+0000 7f067e59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:27.578 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:27.576+0000 7f067e59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:27.586 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:33:27.586 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-11-28T14:33:27.710 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:27.703+0000 7f46a919e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:27.713 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:27.711+0000 7f46a919e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:28.129 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.0 to 1 (10000) 2023-11-28T14:33:28.137 INFO:tasks.rados.rados.0.smithi055.stdout:335: done (2 left) 2023-11-28T14:33:28.137 INFO:tasks.rados.rados.0.smithi055.stdout:336: done (1 left) 2023-11-28T14:33:28.137 INFO:tasks.rados.rados.0.smithi055.stdout:338: done (0 left) 2023-11-28T14:33:28.137 INFO:tasks.rados.rados.0.smithi055.stdout:339: read oid 46 snap -1 2023-11-28T14:33:28.137 INFO:tasks.rados.rados.0.smithi055.stdout:339: expect deleted 2023-11-28T14:33:28.137 INFO:tasks.rados.rados.0.smithi055.stdout:340: snap_create 2023-11-28T14:33:28.147 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-11-28T14:33:28.261 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:28.254+0000 7fb81b86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:28.264 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:28.262+0000 7fb81b86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:29.131 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.1 to 1 (10000) 2023-11-28T14:33:29.139 INFO:tasks.rados.rados.0.smithi055.stdout:339: done (1 left) 2023-11-28T14:33:29.139 INFO:tasks.rados.rados.0.smithi055.stdout:340: done (0 left) 2023-11-28T14:33:29.139 INFO:tasks.rados.rados.0.smithi055.stdout:341: read oid 29 snap 38 2023-11-28T14:33:29.140 INFO:tasks.rados.rados.0.smithi055.stdout:341: expect (ObjNum 167 snap 35 seq_num 167) 2023-11-28T14:33:29.140 INFO:tasks.rados.rados.0.smithi055.stdout:342: snap_create 2023-11-28T14:33:29.149 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-11-28T14:33:29.203 INFO:tasks.rados.rados.0.smithi055.stdout:341: expect (ObjNum 167 snap 35 seq_num 167) 2023-11-28T14:33:29.273 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:29.265+0000 7f744cc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:29.275 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:29.273+0000 7f744cc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:29.663 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:33:29.775 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:29.772+0000 7fc6f34ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:29.778 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:29.776+0000 7fc6f34ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:30.134 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.2 to 1 (10000) 2023-11-28T14:33:30.141 INFO:tasks.rados.rados.0.smithi055.stdout:341: done (1 left) 2023-11-28T14:33:30.141 INFO:tasks.rados.rados.0.smithi055.stdout:342: done (0 left) 2023-11-28T14:33:30.141 INFO:tasks.rados.rados.0.smithi055.stdout:343: snap_create 2023-11-28T14:33:30.148 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-11-28T14:33:30.263 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:30.255+0000 7f0f4155e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:30.265 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:30.264+0000 7f0f4155e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:31.136 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.3 to 1 (10000) 2023-11-28T14:33:31.139 INFO:tasks.rados.rados.0.smithi055.stdout:343: done (0 left) 2023-11-28T14:33:31.139 INFO:tasks.rados.rados.0.smithi055.stdout:344: snap_remove snap 37 2023-11-28T14:33:31.151 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-11-28T14:33:31.152 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:33:31.271 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:31.265+0000 7fc11619e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:31.273 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:31.272+0000 7fc11619e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:31.457 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:33:31.457 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:33:31.469 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2023-11-28T14:33:31.136497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91684077,"num_objects":135,"num_object_clones":81,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":576,"num_read_kb":301301,"num_write":730,"num_write_kb":170326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":969,"ondisk_log_size":969,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":344676,"kb_used_data":229060,"kb_used_omap":0,"kb_used_meta":115584,"kb_avail":377142684,"statfs":{"total":386547056640,"available":386194108416,"internally_reserved":0,"allocated":234557440,"data_stored":255179309,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":118358016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1109262,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6072,"num_write":9,"num_write_kb":2379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005965"},"pg_stats":[{"pgid":"3.f","version":"90'148","reported_seq":287,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.149540+0000","last_change":"2023-11-28T14:33:26.156566+0000","last_active":"2023-11-28T14:33:26.149540+0000","last_peered":"2023-11-28T14:33:26.149540+0000","last_clean":"2023-11-28T14:33:26.149540+0000","last_became_active":"2023-11-28T14:32:09.321992+0000","last_became_peered":"2023-11-28T14:32:09.321992+0000","last_unstale":"2023-11-28T14:33:26.149540+0000","last_undegraded":"2023-11-28T14:33:26.149540+0000","last_fullsized":"2023-11-28T14:33:26.149540+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13445567,"num_objects":21,"num_object_clones":13,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":71,"num_read_kb":47887,"num_write":107,"num_write_kb":31236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'26","reported_seq":183,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.147949+0000","last_change":"2023-11-28T14:33:27.153653+0000","last_active":"2023-11-28T14:33:27.147949+0000","last_peered":"2023-11-28T14:33:27.147949+0000","last_clean":"2023-11-28T14:33:27.147949+0000","last_became_active":"2023-11-28T14:32:09.308848+0000","last_became_peered":"2023-11-28T14:32:09.308848+0000","last_unstale":"2023-11-28T14:33:27.147949+0000","last_undegraded":"2023-11-28T14:33:27.147949+0000","last_fullsized":"2023-11-28T14:33:27.147949+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050454,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":11874,"num_write":19,"num_write_kb":5379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'54","reported_seq":214,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.164248+0000","last_change":"2023-11-28T14:33:27.267190+0000","last_active":"2023-11-28T14:33:28.164248+0000","last_peered":"2023-11-28T14:33:28.164248+0000","last_clean":"2023-11-28T14:33:28.164248+0000","last_became_active":"2023-11-28T14:32:09.337560+0000","last_became_peered":"2023-11-28T14:32:09.337560+0000","last_unstale":"2023-11-28T14:33:28.164248+0000","last_undegraded":"2023-11-28T14:33:28.164248+0000","last_fullsized":"2023-11-28T14:33:28.164248+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428694,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":24018,"num_write":36,"num_write_kb":9320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'49","reported_seq":201,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.150702+0000","last_change":"2023-11-28T14:33:26.155914+0000","last_active":"2023-11-28T14:33:26.150702+0000","last_peered":"2023-11-28T14:33:26.150702+0000","last_clean":"2023-11-28T14:33:26.150702+0000","last_became_active":"2023-11-28T14:32:09.344782+0000","last_became_peered":"2023-11-28T14:32:09.344782+0000","last_unstale":"2023-11-28T14:33:26.150702+0000","last_undegraded":"2023-11-28T14:33:26.150702+0000","last_fullsized":"2023-11-28T14:33:26.150702+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6302592,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":35,"num_read_kb":22807,"num_write":31,"num_write_kb":9129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"92'55","reported_seq":207,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.151105+0000","last_change":"2023-11-28T14:33:28.160903+0000","last_active":"2023-11-28T14:33:28.151105+0000","last_peered":"2023-11-28T14:33:28.151105+0000","last_clean":"2023-11-28T14:33:28.151105+0000","last_became_active":"2023-11-28T14:32:09.326269+0000","last_became_peered":"2023-11-28T14:32:09.326269+0000","last_unstale":"2023-11-28T14:33:28.151105+0000","last_undegraded":"2023-11-28T14:33:28.151105+0000","last_fullsized":"2023-11-28T14:33:28.151105+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5445435,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":44,"num_read_kb":40985,"num_write":38,"num_write_kb":10508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"90'48","reported_seq":191,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.150855+0000","last_change":"2023-11-28T14:33:26.156181+0000","last_active":"2023-11-28T14:33:26.150855+0000","last_peered":"2023-11-28T14:33:26.150855+0000","last_clean":"2023-11-28T14:33:26.150855+0000","last_became_active":"2023-11-28T14:32:09.308846+0000","last_became_peered":"2023-11-28T14:32:09.308846+0000","last_unstale":"2023-11-28T14:33:26.150855+0000","last_undegraded":"2023-11-28T14:33:26.150855+0000","last_fullsized":"2023-11-28T14:33:26.150855+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7747994,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":19,"num_read_kb":5570,"num_write":30,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"80'36","reported_seq":193,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.150029+0000","last_change":"2023-11-28T14:33:26.157823+0000","last_active":"2023-11-28T14:33:26.150029+0000","last_peered":"2023-11-28T14:33:26.150029+0000","last_clean":"2023-11-28T14:33:26.150029+0000","last_became_active":"2023-11-28T14:32:09.316660+0000","last_became_peered":"2023-11-28T14:32:09.316660+0000","last_unstale":"2023-11-28T14:33:26.150029+0000","last_undegraded":"2023-11-28T14:33:26.150029+0000","last_fullsized":"2023-11-28T14:33:26.150029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075714,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":16,"num_read_kb":13969,"num_write":31,"num_write_kb":8807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"92'53","reported_seq":209,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.165023+0000","last_change":"2023-11-28T14:33:28.170641+0000","last_active":"2023-11-28T14:33:28.165023+0000","last_peered":"2023-11-28T14:33:28.165023+0000","last_clean":"2023-11-28T14:33:28.165023+0000","last_became_active":"2023-11-28T14:32:09.358025+0000","last_became_peered":"2023-11-28T14:32:09.358025+0000","last_unstale":"2023-11-28T14:33:28.165023+0000","last_undegraded":"2023-11-28T14:33:28.165023+0000","last_fullsized":"2023-11-28T14:33:28.165023+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226977,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":27,"num_read_kb":18499,"num_write":33,"num_write_kb":9164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":260,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.151933+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:33:28.151933+0000","last_peered":"2023-11-28T14:33:28.151933+0000","last_clean":"2023-11-28T14:33:28.151933+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:33:28.151933+0000","last_undegraded":"2023-11-28T14:33:28.151933+0000","last_fullsized":"2023-11-28T14:33:28.151933+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":135,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.162943+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:33:28.162943+0000","last_peered":"2023-11-28T14:33:28.162943+0000","last_clean":"2023-11-28T14:33:28.162943+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:33:28.162943+0000","last_undegraded":"2023-11-28T14:33:28.162943+0000","last_fullsized":"2023-11-28T14:33:28.162943+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"92'135","reported_seq":291,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.160642+0000","last_change":"2023-11-28T14:33:28.166510+0000","last_active":"2023-11-28T14:33:28.160642+0000","last_peered":"2023-11-28T14:33:28.160642+0000","last_clean":"2023-11-28T14:33:28.160642+0000","last_became_active":"2023-11-28T14:32:09.335805+0000","last_became_peered":"2023-11-28T14:32:09.335805+0000","last_unstale":"2023-11-28T14:33:28.160642+0000","last_undegraded":"2023-11-28T14:33:28.160642+0000","last_fullsized":"2023-11-28T14:33:28.160642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16756513,"num_objects":19,"num_object_clones":11,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":58190,"num_write":93,"num_write_kb":25857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":127,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.148453+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:33:26.148453+0000","last_peered":"2023-11-28T14:33:26.148453+0000","last_clean":"2023-11-28T14:33:26.148453+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:33:26.148453+0000","last_undegraded":"2023-11-28T14:33:26.148453+0000","last_fullsized":"2023-11-28T14:33:26.148453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"84'72","reported_seq":202,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.151006+0000","last_change":"2023-11-28T14:33:27.233014+0000","last_active":"2023-11-28T14:33:28.151006+0000","last_peered":"2023-11-28T14:33:28.151006+0000","last_clean":"2023-11-28T14:33:28.151006+0000","last_became_active":"2023-11-28T14:32:09.350412+0000","last_became_peered":"2023-11-28T14:32:09.350412+0000","last_unstale":"2023-11-28T14:33:28.151006+0000","last_undegraded":"2023-11-28T14:33:28.151006+0000","last_fullsized":"2023-11-28T14:33:28.151006+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588278,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":25,"num_read_kb":8141,"num_write":38,"num_write_kb":11086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.157003+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:33:28.157003+0000","last_peered":"2023-11-28T14:33:28.157003+0000","last_clean":"2023-11-28T14:33:28.157003+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:33:28.157003+0000","last_undegraded":"2023-11-28T14:33:28.157003+0000","last_fullsized":"2023-11-28T14:33:28.157003+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'32","reported_seq":173,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.147560+0000","last_change":"2023-11-28T14:33:27.232343+0000","last_active":"2023-11-28T14:33:28.147560+0000","last_peered":"2023-11-28T14:33:28.147560+0000","last_clean":"2023-11-28T14:33:28.147560+0000","last_became_active":"2023-11-28T14:32:09.343664+0000","last_became_peered":"2023-11-28T14:32:09.343664+0000","last_unstale":"2023-11-28T14:33:28.147560+0000","last_undegraded":"2023-11-28T14:33:28.147560+0000","last_fullsized":"2023-11-28T14:33:28.147560+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932805,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":19,"num_write_kb":5527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":127,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.145877+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:33:26.145877+0000","last_peered":"2023-11-28T14:33:26.145877+0000","last_clean":"2023-11-28T14:33:26.145877+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:33:26.145877+0000","last_undegraded":"2023-11-28T14:33:26.145877+0000","last_fullsized":"2023-11-28T14:33:26.145877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'11","reported_seq":171,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.139864+0000","last_change":"2023-11-28T14:33:27.154193+0000","last_active":"2023-11-28T14:33:27.139864+0000","last_peered":"2023-11-28T14:33:27.139864+0000","last_clean":"2023-11-28T14:33:27.139864+0000","last_became_active":"2023-11-28T14:32:09.311653+0000","last_became_peered":"2023-11-28T14:32:09.311653+0000","last_unstale":"2023-11-28T14:33:27.139864+0000","last_undegraded":"2023-11-28T14:33:27.139864+0000","last_fullsized":"2023-11-28T14:33:27.139864+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2100875,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10016,"num_write":8,"num_write_kb":1910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"74'36","reported_seq":194,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.158968+0000","last_change":"2023-11-28T14:33:27.245124+0000","last_active":"2023-11-28T14:33:28.158968+0000","last_peered":"2023-11-28T14:33:28.158968+0000","last_clean":"2023-11-28T14:33:28.158968+0000","last_became_active":"2023-11-28T14:32:09.316944+0000","last_became_peered":"2023-11-28T14:32:09.316944+0000","last_unstale":"2023-11-28T14:33:28.158968+0000","last_undegraded":"2023-11-28T14:33:28.158968+0000","last_fullsized":"2023-11-28T14:33:28.158968+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672313,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":4786,"num_write":25,"num_write_kb":7544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":131,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.149674+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:33:27.149674+0000","last_peered":"2023-11-28T14:33:27.149674+0000","last_clean":"2023-11-28T14:33:27.149674+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:33:27.149674+0000","last_undegraded":"2023-11-28T14:33:27.149674+0000","last_fullsized":"2023-11-28T14:33:27.149674+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"92'33","reported_seq":190,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.291351+0000","last_change":"2023-11-28T14:33:27.152966+0000","last_active":"2023-11-28T14:33:27.291351+0000","last_peered":"2023-11-28T14:33:27.291351+0000","last_clean":"2023-11-28T14:33:27.291351+0000","last_became_active":"2023-11-28T14:32:09.309510+0000","last_became_peered":"2023-11-28T14:32:09.309510+0000","last_unstale":"2023-11-28T14:33:27.291351+0000","last_undegraded":"2023-11-28T14:33:27.291351+0000","last_fullsized":"2023-11-28T14:33:27.291351+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2974839,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":10725,"num_write":25,"num_write_kb":7305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":127,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.147572+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:33:28.147572+0000","last_peered":"2023-11-28T14:33:28.147572+0000","last_clean":"2023-11-28T14:33:28.147572+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:33:28.147572+0000","last_undegraded":"2023-11-28T14:33:28.147572+0000","last_fullsized":"2023-11-28T14:33:28.147572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"72'42","reported_seq":198,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.156657+0000","last_change":"2023-11-28T14:33:27.244738+0000","last_active":"2023-11-28T14:33:28.156657+0000","last_peered":"2023-11-28T14:33:28.156657+0000","last_clean":"2023-11-28T14:33:28.156657+0000","last_became_active":"2023-11-28T14:32:09.356991+0000","last_became_peered":"2023-11-28T14:32:09.356991+0000","last_unstale":"2023-11-28T14:33:28.156657+0000","last_undegraded":"2023-11-28T14:33:28.156657+0000","last_fullsized":"2023-11-28T14:33:28.156657+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4487469,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":23,"num_read_kb":13848,"num_write":34,"num_write_kb":10390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":131,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.146682+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:33:27.146682+0000","last_peered":"2023-11-28T14:33:27.146682+0000","last_clean":"2023-11-28T14:33:27.146682+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:33:27.146682+0000","last_undegraded":"2023-11-28T14:33:27.146682+0000","last_fullsized":"2023-11-28T14:33:27.146682+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"92'42","reported_seq":188,"reported_epoch":92,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:33:27.149060+0000","last_change":"2023-11-28T14:33:26.540077+0000","last_active":"2023-11-28T14:33:27.149060+0000","last_peered":"2023-11-28T14:33:27.149060+0000","last_clean":"2023-11-28T14:33:27.149060+0000","last_became_active":"2023-11-28T14:32:09.337548+0000","last_became_peered":"2023-11-28T14:32:09.337548+0000","last_unstale":"2023-11-28T14:33:27.149060+0000","last_undegraded":"2023-11-28T14:33:27.149060+0000","last_fullsized":"2023-11-28T14:33:27.149060+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857190,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":21,"num_read_kb":9891,"num_write":26,"num_write_kb":7070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":135,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.153175+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:33:28.153175+0000","last_peered":"2023-11-28T14:33:28.153175+0000","last_clean":"2023-11-28T14:33:28.153175+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:33:28.153175+0000","last_undegraded":"2023-11-28T14:33:28.153175+0000","last_fullsized":"2023-11-28T14:33:28.153175+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":91093709,"num_objects":133,"num_object_clones":81,"num_object_copies":399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":473,"num_read_kb":301210,"num_write":593,"num_write_kb":168432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":872,"ondisk_log_size":872,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89428,"kb_used_data":58444,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94282412,"statfs":{"total":96636764160,"available":96545189888,"internally_reserved":0,"allocated":59846656,"data_stored":64736456,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100000,"kb_used_data":69016,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94271840,"statfs":{"total":96636764160,"available":96534364160,"internally_reserved":0,"allocated":70672384,"data_stored":77676884,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67676,"kb_used_data":40852,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94304164,"statfs":{"total":96636764160,"available":96567463936,"internally_reserved":0,"allocated":41832448,"data_stored":45685109,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87572,"kb_used_data":60748,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94284268,"statfs":{"total":96636764160,"available":96547090432,"internally_reserved":0,"allocated":62205952,"data_stored":67080860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:33:31.470 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:33:31.583 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:31.578+0000 7f682d11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:31.585 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:31.583+0000 7f682d11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:31.770 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:33:31.771 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:33:31.782 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2023-11-28T14:33:31.136497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91684077,"num_objects":135,"num_object_clones":81,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":576,"num_read_kb":301301,"num_write":730,"num_write_kb":170326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":969,"ondisk_log_size":969,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":344676,"kb_used_data":229060,"kb_used_omap":0,"kb_used_meta":115584,"kb_avail":377142684,"statfs":{"total":386547056640,"available":386194108416,"internally_reserved":0,"allocated":234557440,"data_stored":255179309,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":118358016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1109262,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6072,"num_write":9,"num_write_kb":2379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005965"},"pg_stats":[{"pgid":"3.f","version":"90'148","reported_seq":287,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.149540+0000","last_change":"2023-11-28T14:33:26.156566+0000","last_active":"2023-11-28T14:33:26.149540+0000","last_peered":"2023-11-28T14:33:26.149540+0000","last_clean":"2023-11-28T14:33:26.149540+0000","last_became_active":"2023-11-28T14:32:09.321992+0000","last_became_peered":"2023-11-28T14:32:09.321992+0000","last_unstale":"2023-11-28T14:33:26.149540+0000","last_undegraded":"2023-11-28T14:33:26.149540+0000","last_fullsized":"2023-11-28T14:33:26.149540+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13445567,"num_objects":21,"num_object_clones":13,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":71,"num_read_kb":47887,"num_write":107,"num_write_kb":31236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'26","reported_seq":183,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.147949+0000","last_change":"2023-11-28T14:33:27.153653+0000","last_active":"2023-11-28T14:33:27.147949+0000","last_peered":"2023-11-28T14:33:27.147949+0000","last_clean":"2023-11-28T14:33:27.147949+0000","last_became_active":"2023-11-28T14:32:09.308848+0000","last_became_peered":"2023-11-28T14:32:09.308848+0000","last_unstale":"2023-11-28T14:33:27.147949+0000","last_undegraded":"2023-11-28T14:33:27.147949+0000","last_fullsized":"2023-11-28T14:33:27.147949+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050454,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":11874,"num_write":19,"num_write_kb":5379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'54","reported_seq":214,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.164248+0000","last_change":"2023-11-28T14:33:27.267190+0000","last_active":"2023-11-28T14:33:28.164248+0000","last_peered":"2023-11-28T14:33:28.164248+0000","last_clean":"2023-11-28T14:33:28.164248+0000","last_became_active":"2023-11-28T14:32:09.337560+0000","last_became_peered":"2023-11-28T14:32:09.337560+0000","last_unstale":"2023-11-28T14:33:28.164248+0000","last_undegraded":"2023-11-28T14:33:28.164248+0000","last_fullsized":"2023-11-28T14:33:28.164248+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428694,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":24018,"num_write":36,"num_write_kb":9320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'49","reported_seq":201,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.150702+0000","last_change":"2023-11-28T14:33:26.155914+0000","last_active":"2023-11-28T14:33:26.150702+0000","last_peered":"2023-11-28T14:33:26.150702+0000","last_clean":"2023-11-28T14:33:26.150702+0000","last_became_active":"2023-11-28T14:32:09.344782+0000","last_became_peered":"2023-11-28T14:32:09.344782+0000","last_unstale":"2023-11-28T14:33:26.150702+0000","last_undegraded":"2023-11-28T14:33:26.150702+0000","last_fullsized":"2023-11-28T14:33:26.150702+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6302592,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":35,"num_read_kb":22807,"num_write":31,"num_write_kb":9129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"92'55","reported_seq":207,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.151105+0000","last_change":"2023-11-28T14:33:28.160903+0000","last_active":"2023-11-28T14:33:28.151105+0000","last_peered":"2023-11-28T14:33:28.151105+0000","last_clean":"2023-11-28T14:33:28.151105+0000","last_became_active":"2023-11-28T14:32:09.326269+0000","last_became_peered":"2023-11-28T14:32:09.326269+0000","last_unstale":"2023-11-28T14:33:28.151105+0000","last_undegraded":"2023-11-28T14:33:28.151105+0000","last_fullsized":"2023-11-28T14:33:28.151105+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5445435,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":44,"num_read_kb":40985,"num_write":38,"num_write_kb":10508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"90'48","reported_seq":191,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.150855+0000","last_change":"2023-11-28T14:33:26.156181+0000","last_active":"2023-11-28T14:33:26.150855+0000","last_peered":"2023-11-28T14:33:26.150855+0000","last_clean":"2023-11-28T14:33:26.150855+0000","last_became_active":"2023-11-28T14:32:09.308846+0000","last_became_peered":"2023-11-28T14:32:09.308846+0000","last_unstale":"2023-11-28T14:33:26.150855+0000","last_undegraded":"2023-11-28T14:33:26.150855+0000","last_fullsized":"2023-11-28T14:33:26.150855+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7747994,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":19,"num_read_kb":5570,"num_write":30,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"80'36","reported_seq":193,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.150029+0000","last_change":"2023-11-28T14:33:26.157823+0000","last_active":"2023-11-28T14:33:26.150029+0000","last_peered":"2023-11-28T14:33:26.150029+0000","last_clean":"2023-11-28T14:33:26.150029+0000","last_became_active":"2023-11-28T14:32:09.316660+0000","last_became_peered":"2023-11-28T14:32:09.316660+0000","last_unstale":"2023-11-28T14:33:26.150029+0000","last_undegraded":"2023-11-28T14:33:26.150029+0000","last_fullsized":"2023-11-28T14:33:26.150029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075714,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":16,"num_read_kb":13969,"num_write":31,"num_write_kb":8807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"92'53","reported_seq":209,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.165023+0000","last_change":"2023-11-28T14:33:28.170641+0000","last_active":"2023-11-28T14:33:28.165023+0000","last_peered":"2023-11-28T14:33:28.165023+0000","last_clean":"2023-11-28T14:33:28.165023+0000","last_became_active":"2023-11-28T14:32:09.358025+0000","last_became_peered":"2023-11-28T14:32:09.358025+0000","last_unstale":"2023-11-28T14:33:28.165023+0000","last_undegraded":"2023-11-28T14:33:28.165023+0000","last_fullsized":"2023-11-28T14:33:28.165023+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226977,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":27,"num_read_kb":18499,"num_write":33,"num_write_kb":9164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":260,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.151933+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:33:28.151933+0000","last_peered":"2023-11-28T14:33:28.151933+0000","last_clean":"2023-11-28T14:33:28.151933+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:33:28.151933+0000","last_undegraded":"2023-11-28T14:33:28.151933+0000","last_fullsized":"2023-11-28T14:33:28.151933+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":135,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.162943+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:33:28.162943+0000","last_peered":"2023-11-28T14:33:28.162943+0000","last_clean":"2023-11-28T14:33:28.162943+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:33:28.162943+0000","last_undegraded":"2023-11-28T14:33:28.162943+0000","last_fullsized":"2023-11-28T14:33:28.162943+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"92'135","reported_seq":291,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.160642+0000","last_change":"2023-11-28T14:33:28.166510+0000","last_active":"2023-11-28T14:33:28.160642+0000","last_peered":"2023-11-28T14:33:28.160642+0000","last_clean":"2023-11-28T14:33:28.160642+0000","last_became_active":"2023-11-28T14:32:09.335805+0000","last_became_peered":"2023-11-28T14:32:09.335805+0000","last_unstale":"2023-11-28T14:33:28.160642+0000","last_undegraded":"2023-11-28T14:33:28.160642+0000","last_fullsized":"2023-11-28T14:33:28.160642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16756513,"num_objects":19,"num_object_clones":11,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":58190,"num_write":93,"num_write_kb":25857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":127,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.148453+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:33:26.148453+0000","last_peered":"2023-11-28T14:33:26.148453+0000","last_clean":"2023-11-28T14:33:26.148453+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:33:26.148453+0000","last_undegraded":"2023-11-28T14:33:26.148453+0000","last_fullsized":"2023-11-28T14:33:26.148453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"84'72","reported_seq":202,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.151006+0000","last_change":"2023-11-28T14:33:27.233014+0000","last_active":"2023-11-28T14:33:28.151006+0000","last_peered":"2023-11-28T14:33:28.151006+0000","last_clean":"2023-11-28T14:33:28.151006+0000","last_became_active":"2023-11-28T14:32:09.350412+0000","last_became_peered":"2023-11-28T14:32:09.350412+0000","last_unstale":"2023-11-28T14:33:28.151006+0000","last_undegraded":"2023-11-28T14:33:28.151006+0000","last_fullsized":"2023-11-28T14:33:28.151006+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588278,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":25,"num_read_kb":8141,"num_write":38,"num_write_kb":11086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.157003+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:33:28.157003+0000","last_peered":"2023-11-28T14:33:28.157003+0000","last_clean":"2023-11-28T14:33:28.157003+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:33:28.157003+0000","last_undegraded":"2023-11-28T14:33:28.157003+0000","last_fullsized":"2023-11-28T14:33:28.157003+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'32","reported_seq":173,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.147560+0000","last_change":"2023-11-28T14:33:27.232343+0000","last_active":"2023-11-28T14:33:28.147560+0000","last_peered":"2023-11-28T14:33:28.147560+0000","last_clean":"2023-11-28T14:33:28.147560+0000","last_became_active":"2023-11-28T14:32:09.343664+0000","last_became_peered":"2023-11-28T14:32:09.343664+0000","last_unstale":"2023-11-28T14:33:28.147560+0000","last_undegraded":"2023-11-28T14:33:28.147560+0000","last_fullsized":"2023-11-28T14:33:28.147560+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932805,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":19,"num_write_kb":5527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":127,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.145877+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:33:26.145877+0000","last_peered":"2023-11-28T14:33:26.145877+0000","last_clean":"2023-11-28T14:33:26.145877+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:33:26.145877+0000","last_undegraded":"2023-11-28T14:33:26.145877+0000","last_fullsized":"2023-11-28T14:33:26.145877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'11","reported_seq":171,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.139864+0000","last_change":"2023-11-28T14:33:27.154193+0000","last_active":"2023-11-28T14:33:27.139864+0000","last_peered":"2023-11-28T14:33:27.139864+0000","last_clean":"2023-11-28T14:33:27.139864+0000","last_became_active":"2023-11-28T14:32:09.311653+0000","last_became_peered":"2023-11-28T14:32:09.311653+0000","last_unstale":"2023-11-28T14:33:27.139864+0000","last_undegraded":"2023-11-28T14:33:27.139864+0000","last_fullsized":"2023-11-28T14:33:27.139864+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2100875,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10016,"num_write":8,"num_write_kb":1910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"74'36","reported_seq":194,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.158968+0000","last_change":"2023-11-28T14:33:27.245124+0000","last_active":"2023-11-28T14:33:28.158968+0000","last_peered":"2023-11-28T14:33:28.158968+0000","last_clean":"2023-11-28T14:33:28.158968+0000","last_became_active":"2023-11-28T14:32:09.316944+0000","last_became_peered":"2023-11-28T14:32:09.316944+0000","last_unstale":"2023-11-28T14:33:28.158968+0000","last_undegraded":"2023-11-28T14:33:28.158968+0000","last_fullsized":"2023-11-28T14:33:28.158968+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672313,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":4786,"num_write":25,"num_write_kb":7544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":131,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.149674+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:33:27.149674+0000","last_peered":"2023-11-28T14:33:27.149674+0000","last_clean":"2023-11-28T14:33:27.149674+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:33:27.149674+0000","last_undegraded":"2023-11-28T14:33:27.149674+0000","last_fullsized":"2023-11-28T14:33:27.149674+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"92'33","reported_seq":190,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.291351+0000","last_change":"2023-11-28T14:33:27.152966+0000","last_active":"2023-11-28T14:33:27.291351+0000","last_peered":"2023-11-28T14:33:27.291351+0000","last_clean":"2023-11-28T14:33:27.291351+0000","last_became_active":"2023-11-28T14:32:09.309510+0000","last_became_peered":"2023-11-28T14:32:09.309510+0000","last_unstale":"2023-11-28T14:33:27.291351+0000","last_undegraded":"2023-11-28T14:33:27.291351+0000","last_fullsized":"2023-11-28T14:33:27.291351+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2974839,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":10725,"num_write":25,"num_write_kb":7305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":127,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.147572+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:33:28.147572+0000","last_peered":"2023-11-28T14:33:28.147572+0000","last_clean":"2023-11-28T14:33:28.147572+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:33:28.147572+0000","last_undegraded":"2023-11-28T14:33:28.147572+0000","last_fullsized":"2023-11-28T14:33:28.147572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"72'42","reported_seq":198,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.156657+0000","last_change":"2023-11-28T14:33:27.244738+0000","last_active":"2023-11-28T14:33:28.156657+0000","last_peered":"2023-11-28T14:33:28.156657+0000","last_clean":"2023-11-28T14:33:28.156657+0000","last_became_active":"2023-11-28T14:32:09.356991+0000","last_became_peered":"2023-11-28T14:32:09.356991+0000","last_unstale":"2023-11-28T14:33:28.156657+0000","last_undegraded":"2023-11-28T14:33:28.156657+0000","last_fullsized":"2023-11-28T14:33:28.156657+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4487469,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":23,"num_read_kb":13848,"num_write":34,"num_write_kb":10390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":131,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.146682+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:33:27.146682+0000","last_peered":"2023-11-28T14:33:27.146682+0000","last_clean":"2023-11-28T14:33:27.146682+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:33:27.146682+0000","last_undegraded":"2023-11-28T14:33:27.146682+0000","last_fullsized":"2023-11-28T14:33:27.146682+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"92'42","reported_seq":188,"reported_epoch":92,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:33:27.149060+0000","last_change":"2023-11-28T14:33:26.540077+0000","last_active":"2023-11-28T14:33:27.149060+0000","last_peered":"2023-11-28T14:33:27.149060+0000","last_clean":"2023-11-28T14:33:27.149060+0000","last_became_active":"2023-11-28T14:32:09.337548+0000","last_became_peered":"2023-11-28T14:32:09.337548+0000","last_unstale":"2023-11-28T14:33:27.149060+0000","last_undegraded":"2023-11-28T14:33:27.149060+0000","last_fullsized":"2023-11-28T14:33:27.149060+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857190,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":21,"num_read_kb":9891,"num_write":26,"num_write_kb":7070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":135,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.153175+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:33:28.153175+0000","last_peered":"2023-11-28T14:33:28.153175+0000","last_clean":"2023-11-28T14:33:28.153175+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:33:28.153175+0000","last_undegraded":"2023-11-28T14:33:28.153175+0000","last_fullsized":"2023-11-28T14:33:28.153175+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":91093709,"num_objects":133,"num_object_clones":81,"num_object_copies":399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":473,"num_read_kb":301210,"num_write":593,"num_write_kb":168432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":872,"ondisk_log_size":872,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89428,"kb_used_data":58444,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94282412,"statfs":{"total":96636764160,"available":96545189888,"internally_reserved":0,"allocated":59846656,"data_stored":64736456,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100000,"kb_used_data":69016,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94271840,"statfs":{"total":96636764160,"available":96534364160,"internally_reserved":0,"allocated":70672384,"data_stored":77676884,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67676,"kb_used_data":40852,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94304164,"statfs":{"total":96636764160,"available":96567463936,"internally_reserved":0,"allocated":41832448,"data_stored":45685109,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87572,"kb_used_data":60748,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94284268,"statfs":{"total":96636764160,"available":96547090432,"internally_reserved":0,"allocated":62205952,"data_stored":67080860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:33:31.784 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-11-28T14:33:31.784 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:33:31.784 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:33:31.850 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:33:31.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:31.892+0000 7fe4fee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:31.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:31.898+0000 7fe4fee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:31.964 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:31.961+0000 7fbc1851e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:31.967 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:31.965+0000 7fbc1851e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:32.089 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:33:32.090 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:33:32.102 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2023-11-28T14:33:31.136497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91684077,"num_objects":135,"num_object_clones":81,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":576,"num_read_kb":301301,"num_write":730,"num_write_kb":170326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":969,"ondisk_log_size":969,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":344676,"kb_used_data":229060,"kb_used_omap":0,"kb_used_meta":115584,"kb_avail":377142684,"statfs":{"total":386547056640,"available":386194108416,"internally_reserved":0,"allocated":234557440,"data_stored":255179309,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":118358016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1109262,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6072,"num_write":9,"num_write_kb":2379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005965"},"pg_stats":[{"pgid":"3.f","version":"90'148","reported_seq":287,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.149540+0000","last_change":"2023-11-28T14:33:26.156566+0000","last_active":"2023-11-28T14:33:26.149540+0000","last_peered":"2023-11-28T14:33:26.149540+0000","last_clean":"2023-11-28T14:33:26.149540+0000","last_became_active":"2023-11-28T14:32:09.321992+0000","last_became_peered":"2023-11-28T14:32:09.321992+0000","last_unstale":"2023-11-28T14:33:26.149540+0000","last_undegraded":"2023-11-28T14:33:26.149540+0000","last_fullsized":"2023-11-28T14:33:26.149540+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13445567,"num_objects":21,"num_object_clones":13,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":71,"num_read_kb":47887,"num_write":107,"num_write_kb":31236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'26","reported_seq":183,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.147949+0000","last_change":"2023-11-28T14:33:27.153653+0000","last_active":"2023-11-28T14:33:27.147949+0000","last_peered":"2023-11-28T14:33:27.147949+0000","last_clean":"2023-11-28T14:33:27.147949+0000","last_became_active":"2023-11-28T14:32:09.308848+0000","last_became_peered":"2023-11-28T14:32:09.308848+0000","last_unstale":"2023-11-28T14:33:27.147949+0000","last_undegraded":"2023-11-28T14:33:27.147949+0000","last_fullsized":"2023-11-28T14:33:27.147949+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050454,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":11874,"num_write":19,"num_write_kb":5379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'54","reported_seq":214,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.164248+0000","last_change":"2023-11-28T14:33:27.267190+0000","last_active":"2023-11-28T14:33:28.164248+0000","last_peered":"2023-11-28T14:33:28.164248+0000","last_clean":"2023-11-28T14:33:28.164248+0000","last_became_active":"2023-11-28T14:32:09.337560+0000","last_became_peered":"2023-11-28T14:32:09.337560+0000","last_unstale":"2023-11-28T14:33:28.164248+0000","last_undegraded":"2023-11-28T14:33:28.164248+0000","last_fullsized":"2023-11-28T14:33:28.164248+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428694,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":24018,"num_write":36,"num_write_kb":9320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'49","reported_seq":201,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.150702+0000","last_change":"2023-11-28T14:33:26.155914+0000","last_active":"2023-11-28T14:33:26.150702+0000","last_peered":"2023-11-28T14:33:26.150702+0000","last_clean":"2023-11-28T14:33:26.150702+0000","last_became_active":"2023-11-28T14:32:09.344782+0000","last_became_peered":"2023-11-28T14:32:09.344782+0000","last_unstale":"2023-11-28T14:33:26.150702+0000","last_undegraded":"2023-11-28T14:33:26.150702+0000","last_fullsized":"2023-11-28T14:33:26.150702+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6302592,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":35,"num_read_kb":22807,"num_write":31,"num_write_kb":9129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"92'55","reported_seq":207,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.151105+0000","last_change":"2023-11-28T14:33:28.160903+0000","last_active":"2023-11-28T14:33:28.151105+0000","last_peered":"2023-11-28T14:33:28.151105+0000","last_clean":"2023-11-28T14:33:28.151105+0000","last_became_active":"2023-11-28T14:32:09.326269+0000","last_became_peered":"2023-11-28T14:32:09.326269+0000","last_unstale":"2023-11-28T14:33:28.151105+0000","last_undegraded":"2023-11-28T14:33:28.151105+0000","last_fullsized":"2023-11-28T14:33:28.151105+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5445435,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":44,"num_read_kb":40985,"num_write":38,"num_write_kb":10508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"90'48","reported_seq":191,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.150855+0000","last_change":"2023-11-28T14:33:26.156181+0000","last_active":"2023-11-28T14:33:26.150855+0000","last_peered":"2023-11-28T14:33:26.150855+0000","last_clean":"2023-11-28T14:33:26.150855+0000","last_became_active":"2023-11-28T14:32:09.308846+0000","last_became_peered":"2023-11-28T14:32:09.308846+0000","last_unstale":"2023-11-28T14:33:26.150855+0000","last_undegraded":"2023-11-28T14:33:26.150855+0000","last_fullsized":"2023-11-28T14:33:26.150855+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7747994,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":19,"num_read_kb":5570,"num_write":30,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"80'36","reported_seq":193,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.150029+0000","last_change":"2023-11-28T14:33:26.157823+0000","last_active":"2023-11-28T14:33:26.150029+0000","last_peered":"2023-11-28T14:33:26.150029+0000","last_clean":"2023-11-28T14:33:26.150029+0000","last_became_active":"2023-11-28T14:32:09.316660+0000","last_became_peered":"2023-11-28T14:32:09.316660+0000","last_unstale":"2023-11-28T14:33:26.150029+0000","last_undegraded":"2023-11-28T14:33:26.150029+0000","last_fullsized":"2023-11-28T14:33:26.150029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075714,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":16,"num_read_kb":13969,"num_write":31,"num_write_kb":8807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"92'53","reported_seq":209,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.165023+0000","last_change":"2023-11-28T14:33:28.170641+0000","last_active":"2023-11-28T14:33:28.165023+0000","last_peered":"2023-11-28T14:33:28.165023+0000","last_clean":"2023-11-28T14:33:28.165023+0000","last_became_active":"2023-11-28T14:32:09.358025+0000","last_became_peered":"2023-11-28T14:32:09.358025+0000","last_unstale":"2023-11-28T14:33:28.165023+0000","last_undegraded":"2023-11-28T14:33:28.165023+0000","last_fullsized":"2023-11-28T14:33:28.165023+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226977,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":27,"num_read_kb":18499,"num_write":33,"num_write_kb":9164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":260,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.151933+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:33:28.151933+0000","last_peered":"2023-11-28T14:33:28.151933+0000","last_clean":"2023-11-28T14:33:28.151933+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:33:28.151933+0000","last_undegraded":"2023-11-28T14:33:28.151933+0000","last_fullsized":"2023-11-28T14:33:28.151933+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":135,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.162943+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:33:28.162943+0000","last_peered":"2023-11-28T14:33:28.162943+0000","last_clean":"2023-11-28T14:33:28.162943+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:33:28.162943+0000","last_undegraded":"2023-11-28T14:33:28.162943+0000","last_fullsized":"2023-11-28T14:33:28.162943+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"92'135","reported_seq":291,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.160642+0000","last_change":"2023-11-28T14:33:28.166510+0000","last_active":"2023-11-28T14:33:28.160642+0000","last_peered":"2023-11-28T14:33:28.160642+0000","last_clean":"2023-11-28T14:33:28.160642+0000","last_became_active":"2023-11-28T14:32:09.335805+0000","last_became_peered":"2023-11-28T14:32:09.335805+0000","last_unstale":"2023-11-28T14:33:28.160642+0000","last_undegraded":"2023-11-28T14:33:28.160642+0000","last_fullsized":"2023-11-28T14:33:28.160642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16756513,"num_objects":19,"num_object_clones":11,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":58190,"num_write":93,"num_write_kb":25857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":127,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.148453+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:33:26.148453+0000","last_peered":"2023-11-28T14:33:26.148453+0000","last_clean":"2023-11-28T14:33:26.148453+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:33:26.148453+0000","last_undegraded":"2023-11-28T14:33:26.148453+0000","last_fullsized":"2023-11-28T14:33:26.148453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"84'72","reported_seq":202,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.151006+0000","last_change":"2023-11-28T14:33:27.233014+0000","last_active":"2023-11-28T14:33:28.151006+0000","last_peered":"2023-11-28T14:33:28.151006+0000","last_clean":"2023-11-28T14:33:28.151006+0000","last_became_active":"2023-11-28T14:32:09.350412+0000","last_became_peered":"2023-11-28T14:32:09.350412+0000","last_unstale":"2023-11-28T14:33:28.151006+0000","last_undegraded":"2023-11-28T14:33:28.151006+0000","last_fullsized":"2023-11-28T14:33:28.151006+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588278,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":25,"num_read_kb":8141,"num_write":38,"num_write_kb":11086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.157003+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:33:28.157003+0000","last_peered":"2023-11-28T14:33:28.157003+0000","last_clean":"2023-11-28T14:33:28.157003+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:33:28.157003+0000","last_undegraded":"2023-11-28T14:33:28.157003+0000","last_fullsized":"2023-11-28T14:33:28.157003+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'32","reported_seq":173,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.147560+0000","last_change":"2023-11-28T14:33:27.232343+0000","last_active":"2023-11-28T14:33:28.147560+0000","last_peered":"2023-11-28T14:33:28.147560+0000","last_clean":"2023-11-28T14:33:28.147560+0000","last_became_active":"2023-11-28T14:32:09.343664+0000","last_became_peered":"2023-11-28T14:32:09.343664+0000","last_unstale":"2023-11-28T14:33:28.147560+0000","last_undegraded":"2023-11-28T14:33:28.147560+0000","last_fullsized":"2023-11-28T14:33:28.147560+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932805,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":19,"num_write_kb":5527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":127,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-28T14:33:26.145877+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:33:26.145877+0000","last_peered":"2023-11-28T14:33:26.145877+0000","last_clean":"2023-11-28T14:33:26.145877+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:33:26.145877+0000","last_undegraded":"2023-11-28T14:33:26.145877+0000","last_fullsized":"2023-11-28T14:33:26.145877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'11","reported_seq":171,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.139864+0000","last_change":"2023-11-28T14:33:27.154193+0000","last_active":"2023-11-28T14:33:27.139864+0000","last_peered":"2023-11-28T14:33:27.139864+0000","last_clean":"2023-11-28T14:33:27.139864+0000","last_became_active":"2023-11-28T14:32:09.311653+0000","last_became_peered":"2023-11-28T14:32:09.311653+0000","last_unstale":"2023-11-28T14:33:27.139864+0000","last_undegraded":"2023-11-28T14:33:27.139864+0000","last_fullsized":"2023-11-28T14:33:27.139864+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2100875,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10016,"num_write":8,"num_write_kb":1910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"74'36","reported_seq":194,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.158968+0000","last_change":"2023-11-28T14:33:27.245124+0000","last_active":"2023-11-28T14:33:28.158968+0000","last_peered":"2023-11-28T14:33:28.158968+0000","last_clean":"2023-11-28T14:33:28.158968+0000","last_became_active":"2023-11-28T14:32:09.316944+0000","last_became_peered":"2023-11-28T14:32:09.316944+0000","last_unstale":"2023-11-28T14:33:28.158968+0000","last_undegraded":"2023-11-28T14:33:28.158968+0000","last_fullsized":"2023-11-28T14:33:28.158968+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672313,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":4786,"num_write":25,"num_write_kb":7544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":131,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.149674+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:33:27.149674+0000","last_peered":"2023-11-28T14:33:27.149674+0000","last_clean":"2023-11-28T14:33:27.149674+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:33:27.149674+0000","last_undegraded":"2023-11-28T14:33:27.149674+0000","last_fullsized":"2023-11-28T14:33:27.149674+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"92'33","reported_seq":190,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.291351+0000","last_change":"2023-11-28T14:33:27.152966+0000","last_active":"2023-11-28T14:33:27.291351+0000","last_peered":"2023-11-28T14:33:27.291351+0000","last_clean":"2023-11-28T14:33:27.291351+0000","last_became_active":"2023-11-28T14:32:09.309510+0000","last_became_peered":"2023-11-28T14:32:09.309510+0000","last_unstale":"2023-11-28T14:33:27.291351+0000","last_undegraded":"2023-11-28T14:33:27.291351+0000","last_fullsized":"2023-11-28T14:33:27.291351+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2974839,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":10725,"num_write":25,"num_write_kb":7305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":127,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.147572+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:33:28.147572+0000","last_peered":"2023-11-28T14:33:28.147572+0000","last_clean":"2023-11-28T14:33:28.147572+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:33:28.147572+0000","last_undegraded":"2023-11-28T14:33:28.147572+0000","last_fullsized":"2023-11-28T14:33:28.147572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"72'42","reported_seq":198,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.156657+0000","last_change":"2023-11-28T14:33:27.244738+0000","last_active":"2023-11-28T14:33:28.156657+0000","last_peered":"2023-11-28T14:33:28.156657+0000","last_clean":"2023-11-28T14:33:28.156657+0000","last_became_active":"2023-11-28T14:32:09.356991+0000","last_became_peered":"2023-11-28T14:32:09.356991+0000","last_unstale":"2023-11-28T14:33:28.156657+0000","last_undegraded":"2023-11-28T14:33:28.156657+0000","last_fullsized":"2023-11-28T14:33:28.156657+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4487469,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":23,"num_read_kb":13848,"num_write":34,"num_write_kb":10390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":131,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-11-28T14:33:27.146682+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:33:27.146682+0000","last_peered":"2023-11-28T14:33:27.146682+0000","last_clean":"2023-11-28T14:33:27.146682+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:33:27.146682+0000","last_undegraded":"2023-11-28T14:33:27.146682+0000","last_fullsized":"2023-11-28T14:33:27.146682+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"92'42","reported_seq":188,"reported_epoch":92,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:33:27.149060+0000","last_change":"2023-11-28T14:33:26.540077+0000","last_active":"2023-11-28T14:33:27.149060+0000","last_peered":"2023-11-28T14:33:27.149060+0000","last_clean":"2023-11-28T14:33:27.149060+0000","last_became_active":"2023-11-28T14:32:09.337548+0000","last_became_peered":"2023-11-28T14:32:09.337548+0000","last_unstale":"2023-11-28T14:33:27.149060+0000","last_undegraded":"2023-11-28T14:33:27.149060+0000","last_fullsized":"2023-11-28T14:33:27.149060+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857190,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":21,"num_read_kb":9891,"num_write":26,"num_write_kb":7070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":135,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-28T14:33:28.153175+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:33:28.153175+0000","last_peered":"2023-11-28T14:33:28.153175+0000","last_clean":"2023-11-28T14:33:28.153175+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:33:28.153175+0000","last_undegraded":"2023-11-28T14:33:28.153175+0000","last_fullsized":"2023-11-28T14:33:28.153175+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":91093709,"num_objects":133,"num_object_clones":81,"num_object_copies":399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":473,"num_read_kb":301210,"num_write":593,"num_write_kb":168432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":872,"ondisk_log_size":872,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89428,"kb_used_data":58444,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94282412,"statfs":{"total":96636764160,"available":96545189888,"internally_reserved":0,"allocated":59846656,"data_stored":64736456,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100000,"kb_used_data":69016,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94271840,"statfs":{"total":96636764160,"available":96534364160,"internally_reserved":0,"allocated":70672384,"data_stored":77676884,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67676,"kb_used_data":40852,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94304164,"statfs":{"total":96636764160,"available":96567463936,"internally_reserved":0,"allocated":41832448,"data_stored":45685109,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87572,"kb_used_data":60748,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94284268,"statfs":{"total":96636764160,"available":96547090432,"internally_reserved":0,"allocated":62205952,"data_stored":67080860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:33:32.143 INFO:tasks.rados.rados.0.smithi055.stdout:344: done (0 left) 2023-11-28T14:33:32.144 INFO:tasks.rados.rados.0.smithi055.stdout:345: snap_create 2023-11-28T14:33:33.141 INFO:tasks.rados.rados.0.smithi055.stdout:345: done (0 left) 2023-11-28T14:33:33.142 INFO:tasks.rados.rados.0.smithi055.stdout:346: write (excl) oid 4 current snap is 45 2023-11-28T14:33:33.142 INFO:tasks.rados.rados.0.smithi055.stdout:346: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2023-11-28T14:33:33.148 INFO:tasks.rados.rados.0.smithi055.stdout:346: writing smithi05583006-4 from 608453 to 1154143 tid 1 2023-11-28T14:33:33.153 INFO:tasks.rados.rados.0.smithi055.stdout:346: writing smithi05583006-4 from 1624865 to 2080593 tid 2 2023-11-28T14:33:33.158 INFO:tasks.rados.rados.0.smithi055.stdout:346: writing smithi05583006-4 from 2632598 to 3008453 tid 3 2023-11-28T14:33:33.158 INFO:tasks.rados.rados.0.smithi055.stdout:347: write (excl) oid 22 current snap is 45 2023-11-28T14:33:33.158 INFO:tasks.rados.rados.0.smithi055.stdout:347: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2023-11-28T14:33:33.165 INFO:tasks.rados.rados.0.smithi055.stdout:347: writing smithi05583006-22 from 625260 to 1246199 tid 1 2023-11-28T14:33:33.170 INFO:tasks.rados.rados.0.smithi055.stdout:347: writing smithi05583006-22 from 1683347 to 2282733 tid 2 2023-11-28T14:33:33.175 INFO:tasks.rados.rados.0.smithi055.stdout:347: writing smithi05583006-22 from 2943668 to 3025260 tid 3 2023-11-28T14:33:33.175 INFO:tasks.rados.rados.0.smithi055.stdout:348: write oid 34 current snap is 45 2023-11-28T14:33:33.175 INFO:tasks.rados.rados.0.smithi055.stdout:348: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2023-11-28T14:33:33.182 INFO:tasks.rados.rados.0.smithi055.stdout:348: writing smithi05583006-34 from 642067 to 1338255 tid 1 2023-11-28T14:33:33.188 INFO:tasks.rados.rados.0.smithi055.stdout:348: writing smithi05583006-34 from 2025476 to 2484873 tid 2 2023-11-28T14:33:33.191 INFO:tasks.rados.rados.0.smithi055.stdout:348: writing smithi05583006-34 from 2971091 to 3042067 tid 3 2023-11-28T14:33:33.192 INFO:tasks.rados.rados.0.smithi055.stdout:349: write oid 15 current snap is 45 2023-11-28T14:33:33.192 INFO:tasks.rados.rados.0.smithi055.stdout:349: seq_num 182 ranges {658874=771437,2083958=603055} 2023-11-28T14:33:33.199 INFO:tasks.rados.rados.0.smithi055.stdout:349: writing smithi05583006-15 from 658874 to 1430311 tid 1 2023-11-28T14:33:33.206 INFO:tasks.rados.rados.0.smithi055.stdout:349: writing smithi05583006-15 from 2083958 to 2687013 tid 2 2023-11-28T14:33:33.206 INFO:tasks.rados.rados.0.smithi055.stdout:350: read oid 10 snap 31 2023-11-28T14:33:33.206 INFO:tasks.rados.rados.0.smithi055.stdout:350: expect (ObjNum 154 snap 28 seq_num 154) 2023-11-28T14:33:33.206 INFO:tasks.rados.rados.0.smithi055.stdout:351: snap_create 2023-11-28T14:33:33.229 INFO:tasks.rados.rados.0.smithi055.stdout:350: expect (ObjNum 154 snap 28 seq_num 154) 2023-11-28T14:33:33.377 INFO:tasks.rados.rados.0.smithi055.stdout:346: finishing write tid 1 to smithi05583006-4 2023-11-28T14:33:33.377 INFO:tasks.rados.rados.0.smithi055.stdout:348: finishing write tid 1 to smithi05583006-34 2023-11-28T14:33:33.378 INFO:tasks.rados.rados.0.smithi055.stdout:347: finishing write tid 1 to smithi05583006-22 2023-11-28T14:33:33.378 INFO:tasks.rados.rados.0.smithi055.stdout:346: finishing write tid 2 to smithi05583006-4 2023-11-28T14:33:33.378 INFO:tasks.rados.rados.0.smithi055.stdout:349: finishing write tid 1 to smithi05583006-15 2023-11-28T14:33:33.378 INFO:tasks.rados.rados.0.smithi055.stdout:346: finishing write tid 3 to smithi05583006-4 2023-11-28T14:33:33.378 INFO:tasks.rados.rados.0.smithi055.stdout:348: finishing write tid 2 to smithi05583006-34 2023-11-28T14:33:33.378 INFO:tasks.rados.rados.0.smithi055.stdout:347: finishing write tid 2 to smithi05583006-22 2023-11-28T14:33:33.378 INFO:tasks.rados.rados.0.smithi055.stdout:349: finishing write tid 2 to smithi05583006-15 2023-11-28T14:33:33.378 INFO:tasks.rados.rados.0.smithi055.stdout:348: finishing write tid 3 to smithi05583006-34 2023-11-28T14:33:33.378 INFO:tasks.rados.rados.0.smithi055.stdout:347: finishing write tid 3 to smithi05583006-22 2023-11-28T14:33:33.378 INFO:tasks.rados.rados.0.smithi055.stdout:346: finishing write tid 4 to smithi05583006-4 2023-11-28T14:33:33.378 INFO:tasks.rados.rados.0.smithi055.stdout:346: finishing write tid 5 to smithi05583006-4 2023-11-28T14:33:33.378 INFO:tasks.rados.rados.0.smithi055.stdout:346: oid 4 updating version 0 to 38 2023-11-28T14:33:33.379 INFO:tasks.rados.rados.0.smithi055.stdout:346: oid 4 updating version 38 to 39 2023-11-28T14:33:33.379 INFO:tasks.rados.rados.0.smithi055.stdout:346: oid 4 updating version 39 to 40 2023-11-28T14:33:33.379 INFO:tasks.rados.rados.0.smithi055.stdout:346: oid 4 updating version 40 to 41 2023-11-28T14:33:33.379 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 4 v 41 (ObjNum 179 snap 45 seq_num 179) dirty exists 2023-11-28T14:33:33.379 INFO:tasks.rados.rados.0.smithi055.stdout:346: left oid 4 (ObjNum 179 snap 45 seq_num 179) 2023-11-28T14:33:33.379 INFO:tasks.rados.rados.0.smithi055.stdout:349: finishing write tid 3 to smithi05583006-15 2023-11-28T14:33:33.379 INFO:tasks.rados.rados.0.smithi055.stdout:349: finishing write tid 4 to smithi05583006-15 2023-11-28T14:33:33.379 INFO:tasks.rados.rados.0.smithi055.stdout:349: oid 15 updating version 0 to 50 2023-11-28T14:33:33.379 INFO:tasks.rados.rados.0.smithi055.stdout:349: oid 15 updating version 50 to 52 2023-11-28T14:33:33.379 INFO:tasks.rados.rados.0.smithi055.stdout:349: oid 15 version 52 is already newer than 51 2023-11-28T14:33:33.380 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 15 v 52 (ObjNum 182 snap 45 seq_num 182) dirty exists 2023-11-28T14:33:33.380 INFO:tasks.rados.rados.0.smithi055.stdout:349: left oid 15 (ObjNum 182 snap 45 seq_num 182) 2023-11-28T14:33:33.380 INFO:tasks.rados.rados.0.smithi055.stdout:347: finishing write tid 4 to smithi05583006-22 2023-11-28T14:33:33.380 INFO:tasks.rados.rados.0.smithi055.stdout:348: finishing write tid 4 to smithi05583006-34 2023-11-28T14:33:33.380 INFO:tasks.rados.rados.0.smithi055.stdout:348: finishing write tid 5 to smithi05583006-34 2023-11-28T14:33:33.380 INFO:tasks.rados.rados.0.smithi055.stdout:348: oid 34 updating version 0 to 44 2023-11-28T14:33:33.380 INFO:tasks.rados.rados.0.smithi055.stdout:348: oid 34 updating version 44 to 45 2023-11-28T14:33:33.380 INFO:tasks.rados.rados.0.smithi055.stdout:348: oid 34 updating version 45 to 46 2023-11-28T14:33:33.380 INFO:tasks.rados.rados.0.smithi055.stdout:348: oid 34 version 46 is already newer than 43 2023-11-28T14:33:33.380 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 34 v 46 (ObjNum 181 snap 45 seq_num 181) dirty exists 2023-11-28T14:33:33.381 INFO:tasks.rados.rados.0.smithi055.stdout:348: left oid 34 (ObjNum 181 snap 45 seq_num 181) 2023-11-28T14:33:33.381 INFO:tasks.rados.rados.0.smithi055.stdout:347: finishing write tid 5 to smithi05583006-22 2023-11-28T14:33:33.381 INFO:tasks.rados.rados.0.smithi055.stdout:347: oid 22 updating version 0 to 153 2023-11-28T14:33:33.381 INFO:tasks.rados.rados.0.smithi055.stdout:347: oid 22 updating version 153 to 154 2023-11-28T14:33:33.381 INFO:tasks.rados.rados.0.smithi055.stdout:347: oid 22 version 154 is already newer than 152 2023-11-28T14:33:33.381 INFO:tasks.rados.rados.0.smithi055.stdout:347: oid 22 updating version 154 to 155 2023-11-28T14:33:33.381 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 22 v 155 (ObjNum 180 snap 45 seq_num 180) dirty exists 2023-11-28T14:33:33.381 INFO:tasks.rados.rados.0.smithi055.stdout:347: left oid 22 (ObjNum 180 snap 45 seq_num 180) 2023-11-28T14:33:34.039 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:33:34.163 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:34.161+0000 7f03bca2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:34.167 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:34.166+0000 7f03bca2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:34.181 INFO:tasks.rados.rados.0.smithi055.stdout:346: done (5 left) 2023-11-28T14:33:34.181 INFO:tasks.rados.rados.0.smithi055.stdout:347: done (4 left) 2023-11-28T14:33:34.182 INFO:tasks.rados.rados.0.smithi055.stdout:348: done (3 left) 2023-11-28T14:33:34.182 INFO:tasks.rados.rados.0.smithi055.stdout:349: done (2 left) 2023-11-28T14:33:34.182 INFO:tasks.rados.rados.0.smithi055.stdout:350: done (1 left) 2023-11-28T14:33:34.182 INFO:tasks.rados.rados.0.smithi055.stdout:351: done (0 left) 2023-11-28T14:33:34.182 INFO:tasks.rados.rados.0.smithi055.stdout:352: read oid 27 snap 34 2023-11-28T14:33:34.182 INFO:tasks.rados.rados.0.smithi055.stdout:352: expect (ObjNum 26 snap 0 seq_num 26) 2023-11-28T14:33:34.182 INFO:tasks.rados.rados.0.smithi055.stdout:353: write (excl) oid 15 current snap is 46 2023-11-28T14:33:34.182 INFO:tasks.rados.rados.0.smithi055.stdout:353: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2023-11-28T14:33:34.189 INFO:tasks.rados.rados.0.smithi055.stdout:353: writing smithi05583006-15 from 675681 to 1238720 tid 1 2023-11-28T14:33:34.199 INFO:tasks.rados.rados.0.smithi055.stdout:353: writing smithi05583006-15 from 1858793 to 2605506 tid 2 2023-11-28T14:33:34.202 INFO:tasks.rados.rados.0.smithi055.stdout:353: writing smithi05583006-15 from 3025937 to 3075681 tid 3 2023-11-28T14:33:34.202 INFO:tasks.rados.rados.0.smithi055.stdout:354: write oid 16 current snap is 46 2023-11-28T14:33:34.202 INFO:tasks.rados.rados.0.smithi055.stdout:354: seq_num 184 ranges {692488=638288,1917275=606724} 2023-11-28T14:33:34.210 INFO:tasks.rados.rados.0.smithi055.stdout:354: writing smithi05583006-16 from 692488 to 1330776 tid 1 2023-11-28T14:33:34.217 INFO:tasks.rados.rados.0.smithi055.stdout:354: writing smithi05583006-16 from 1917275 to 2523999 tid 2 2023-11-28T14:33:34.217 INFO:tasks.rados.rados.0.smithi055.stdout:355: read oid 39 snap -1 2023-11-28T14:33:34.217 INFO:tasks.rados.rados.0.smithi055.stdout:355: expect (ObjNum 130 snap 18 seq_num 130) 2023-11-28T14:33:34.217 INFO:tasks.rados.rados.0.smithi055.stdout:356: snap_create 2023-11-28T14:33:34.269 INFO:tasks.rados.rados.0.smithi055.stdout:355: expect (ObjNum 130 snap 18 seq_num 130) 2023-11-28T14:33:34.373 INFO:tasks.rados.rados.0.smithi055.stdout:352: expect (ObjNum 26 snap 0 seq_num 26) 2023-11-28T14:33:34.466 INFO:tasks.rados.rados.0.smithi055.stdout:354: finishing write tid 1 to smithi05583006-16 2023-11-28T14:33:34.466 INFO:tasks.rados.rados.0.smithi055.stdout:353: finishing write tid 1 to smithi05583006-15 2023-11-28T14:33:34.466 INFO:tasks.rados.rados.0.smithi055.stdout:353: finishing write tid 2 to smithi05583006-15 2023-11-28T14:33:34.466 INFO:tasks.rados.rados.0.smithi055.stdout:354: finishing write tid 2 to smithi05583006-16 2023-11-28T14:33:34.466 INFO:tasks.rados.rados.0.smithi055.stdout:353: finishing write tid 3 to smithi05583006-15 2023-11-28T14:33:34.466 INFO:tasks.rados.rados.0.smithi055.stdout:354: finishing write tid 3 to smithi05583006-16 2023-11-28T14:33:34.466 INFO:tasks.rados.rados.0.smithi055.stdout:354: finishing write tid 4 to smithi05583006-16 2023-11-28T14:33:34.467 INFO:tasks.rados.rados.0.smithi055.stdout:354: oid 16 updating version 0 to 141 2023-11-28T14:33:34.467 INFO:tasks.rados.rados.0.smithi055.stdout:354: oid 16 version 141 is already newer than 140 2023-11-28T14:33:34.467 INFO:tasks.rados.rados.0.smithi055.stdout:354: oid 16 version 141 is already newer than 139 2023-11-28T14:33:34.467 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 16 v 141 (ObjNum 184 snap 46 seq_num 184) dirty exists 2023-11-28T14:33:34.467 INFO:tasks.rados.rados.0.smithi055.stdout:354: left oid 16 (ObjNum 184 snap 46 seq_num 184) 2023-11-28T14:33:34.467 INFO:tasks.rados.rados.0.smithi055.stdout:353: finishing write tid 4 to smithi05583006-15 2023-11-28T14:33:34.467 INFO:tasks.rados.rados.0.smithi055.stdout:353: finishing write tid 5 to smithi05583006-15 2023-11-28T14:33:34.467 INFO:tasks.rados.rados.0.smithi055.stdout:353: oid 15 updating version 0 to 55 2023-11-28T14:33:34.467 INFO:tasks.rados.rados.0.smithi055.stdout:353: oid 15 updating version 55 to 57 2023-11-28T14:33:34.467 INFO:tasks.rados.rados.0.smithi055.stdout:353: oid 15 version 57 is already newer than 56 2023-11-28T14:33:34.468 INFO:tasks.rados.rados.0.smithi055.stdout:353: oid 15 version 57 is already newer than 54 2023-11-28T14:33:34.468 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 15 v 57 (ObjNum 183 snap 46 seq_num 183) dirty exists 2023-11-28T14:33:34.468 INFO:tasks.rados.rados.0.smithi055.stdout:353: left oid 15 (ObjNum 183 snap 46 seq_num 183) 2023-11-28T14:33:35.132 INFO:tasks.rados.rados.0.smithi055.stdout:352: done (4 left) 2023-11-28T14:33:35.132 INFO:tasks.rados.rados.0.smithi055.stdout:353: done (3 left) 2023-11-28T14:33:35.132 INFO:tasks.rados.rados.0.smithi055.stdout:354: done (2 left) 2023-11-28T14:33:35.133 INFO:tasks.rados.rados.0.smithi055.stdout:355: done (1 left) 2023-11-28T14:33:35.133 INFO:tasks.rados.rados.0.smithi055.stdout:356: done (0 left) 2023-11-28T14:33:35.133 INFO:tasks.rados.rados.0.smithi055.stdout:357: snap_remove snap 35 2023-11-28T14:33:36.133 INFO:tasks.rados.rados.0.smithi055.stdout:357: done (0 left) 2023-11-28T14:33:36.133 INFO:tasks.rados.rados.0.smithi055.stdout:358: write oid 38 current snap is 47 2023-11-28T14:33:36.133 INFO:tasks.rados.rados.0.smithi055.stdout:358: seq_num 185 ranges {709295=713537,1859404=750382} 2023-11-28T14:33:36.141 INFO:tasks.rados.rados.0.smithi055.stdout:358: writing smithi05583006-38 from 709295 to 1422832 tid 1 2023-11-28T14:33:36.147 INFO:tasks.rados.rados.0.smithi055.stdout:358: writing smithi05583006-38 from 1859404 to 2609786 tid 2 2023-11-28T14:33:36.148 INFO:tasks.rados.rados.0.smithi055.stdout:359: read oid 40 snap -1 2023-11-28T14:33:36.148 INFO:tasks.rados.rados.0.smithi055.stdout:359: expect (ObjNum 122 snap 18 seq_num 122) 2023-11-28T14:33:36.148 INFO:tasks.rados.rados.0.smithi055.stdout:360: delete oid 22 current snap is 47 2023-11-28T14:33:36.234 INFO:tasks.rados.rados.0.smithi055.stdout:359: expect (ObjNum 122 snap 18 seq_num 122) 2023-11-28T14:33:36.334 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:33:36.355 INFO:tasks.rados.rados.0.smithi055.stdout:358: finishing write tid 1 to smithi05583006-38 2023-11-28T14:33:36.355 INFO:tasks.rados.rados.0.smithi055.stdout:358: finishing write tid 2 to smithi05583006-38 2023-11-28T14:33:36.355 INFO:tasks.rados.rados.0.smithi055.stdout:359: done (2 left) 2023-11-28T14:33:36.355 INFO:tasks.rados.rados.0.smithi055.stdout:360: done (1 left) 2023-11-28T14:33:36.356 INFO:tasks.rados.rados.0.smithi055.stdout:361: read oid 21 snap -1 2023-11-28T14:33:36.356 INFO:tasks.rados.rados.0.smithi055.stdout:361: expect deleted 2023-11-28T14:33:36.356 INFO:tasks.rados.rados.0.smithi055.stdout:362: snap_remove snap 45 2023-11-28T14:33:36.444 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:36.442+0000 7fe2160f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:36.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:36.446+0000 7fe2160f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:37.103 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:33:37.103 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:33:37.104 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:33:37.141 INFO:tasks.rados.rados.0.smithi055.stdout:362: done (2 left) 2023-11-28T14:33:37.141 INFO:tasks.rados.rados.0.smithi055.stdout:363: write (excl) oid 8 current snap is 47 2023-11-28T14:33:37.142 INFO:tasks.rados.rados.0.smithi055.stdout:363: seq_num 187 ranges {742909=464035,1976368=754051} 2023-11-28T14:33:37.149 INFO:tasks.rados.rados.0.smithi055.stdout:363: writing smithi05583006-8 from 742909 to 1206944 tid 1 2023-11-28T14:33:37.161 INFO:tasks.rados.rados.0.smithi055.stdout:363: writing smithi05583006-8 from 1976368 to 2730419 tid 2 2023-11-28T14:33:37.161 INFO:tasks.rados.rados.0.smithi055.stdout:364: read oid 27 snap -1 2023-11-28T14:33:37.161 INFO:tasks.rados.rados.0.smithi055.stdout:364: expect (ObjNum 170 snap 36 seq_num 170) 2023-11-28T14:33:37.161 INFO:tasks.rados.rados.0.smithi055.stdout:365: read oid 33 snap -1 2023-11-28T14:33:37.161 INFO:tasks.rados.rados.0.smithi055.stdout:365: expect (ObjNum 174 snap 40 seq_num 174) 2023-11-28T14:33:37.161 INFO:tasks.rados.rados.0.smithi055.stdout:366: read oid 34 snap -1 2023-11-28T14:33:37.161 INFO:tasks.rados.rados.0.smithi055.stdout:366: expect (ObjNum 181 snap 45 seq_num 181) 2023-11-28T14:33:37.161 INFO:tasks.rados.rados.0.smithi055.stdout:367: delete oid 42 current snap is 47 2023-11-28T14:33:37.162 INFO:tasks.rados.rados.0.smithi055.stdout:358: finishing write tid 3 to smithi05583006-38 2023-11-28T14:33:37.162 INFO:tasks.rados.rados.0.smithi055.stdout:358: finishing write tid 4 to smithi05583006-38 2023-11-28T14:33:37.162 INFO:tasks.rados.rados.0.smithi055.stdout:358: oid 38 updating version 0 to 57 2023-11-28T14:33:37.162 INFO:tasks.rados.rados.0.smithi055.stdout:358: oid 38 version 57 is already newer than 57 2023-11-28T14:33:37.162 INFO:tasks.rados.rados.0.smithi055.stdout:358: oid 38 version 57 is already newer than 55 2023-11-28T14:33:37.162 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 38 v 57 (ObjNum 185 snap 47 seq_num 185) dirty exists 2023-11-28T14:33:37.162 INFO:tasks.rados.rados.0.smithi055.stdout:358: left oid 38 (ObjNum 185 snap 47 seq_num 185) 2023-11-28T14:33:37.236 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:37.231+0000 7f421202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:37.239 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:37.239+0000 7f421202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:37.246 INFO:tasks.rados.rados.0.smithi055.stdout:364: expect (ObjNum 170 snap 36 seq_num 170) 2023-11-28T14:33:37.403 INFO:tasks.rados.rados.0.smithi055.stdout:363: finishing write tid 1 to smithi05583006-8 2023-11-28T14:33:37.403 INFO:tasks.rados.rados.0.smithi055.stdout:366: expect (ObjNum 181 snap 45 seq_num 181) 2023-11-28T14:33:37.434 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:33:37.437 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:33:37.446 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2023-11-28T14:33:37.135029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95062284,"num_objects":137,"num_object_clones":82,"num_object_copies":411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":587,"num_read_kb":312039,"num_write":740,"num_write_kb":172872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":990,"ondisk_log_size":990,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":345980,"kb_used_data":230364,"kb_used_omap":0,"kb_used_meta":115584,"kb_avail":377141380,"statfs":{"total":386547056640,"available":386192773120,"internally_reserved":0,"allocated":235892736,"data_stored":256860728,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":118358016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3378207,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":6486,"num_write":10,"num_write_kb":2546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.995990"},"pg_stats":[{"pgid":"3.f","version":"92'150","reported_seq":298,"reported_epoch":97,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:33:32.500517+0000","last_change":"2023-11-28T14:33:32.500517+0000","last_active":"2023-11-28T14:33:32.500517+0000","last_peered":"2023-11-28T14:33:32.500517+0000","last_clean":"2023-11-28T14:33:32.500517+0000","last_became_active":"2023-11-28T14:32:09.321992+0000","last_became_peered":"2023-11-28T14:32:09.321992+0000","last_unstale":"2023-11-28T14:33:32.500517+0000","last_undegraded":"2023-11-28T14:33:32.500517+0000","last_fullsized":"2023-11-28T14:33:32.500517+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13445567,"num_objects":21,"num_object_clones":13,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":71,"num_read_kb":47887,"num_write":107,"num_write_kb":31236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'26","reported_seq":190,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-11-28T14:33:32.164021+0000","last_change":"2023-11-28T14:33:32.172276+0000","last_active":"2023-11-28T14:33:32.164021+0000","last_peered":"2023-11-28T14:33:32.164021+0000","last_clean":"2023-11-28T14:33:32.164021+0000","last_became_active":"2023-11-28T14:32:09.308848+0000","last_became_peered":"2023-11-28T14:32:09.308848+0000","last_unstale":"2023-11-28T14:33:32.164021+0000","last_undegraded":"2023-11-28T14:33:32.164021+0000","last_fullsized":"2023-11-28T14:33:32.164021+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050454,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":11874,"num_write":19,"num_write_kb":5379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'54","reported_seq":223,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.165072+0000","last_change":"2023-11-28T14:33:32.516199+0000","last_active":"2023-11-28T14:33:33.165072+0000","last_peered":"2023-11-28T14:33:33.165072+0000","last_clean":"2023-11-28T14:33:33.165072+0000","last_became_active":"2023-11-28T14:32:09.337560+0000","last_became_peered":"2023-11-28T14:32:09.337560+0000","last_unstale":"2023-11-28T14:33:33.165072+0000","last_undegraded":"2023-11-28T14:33:33.165072+0000","last_fullsized":"2023-11-28T14:33:33.165072+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428694,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":24018,"num_write":36,"num_write_kb":9320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"92'51","reported_seq":210,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-11-28T14:33:30.768034+0000","last_change":"2023-11-28T14:33:27.675177+0000","last_active":"2023-11-28T14:33:30.768034+0000","last_peered":"2023-11-28T14:33:30.768034+0000","last_clean":"2023-11-28T14:33:30.768034+0000","last_became_active":"2023-11-28T14:32:09.344782+0000","last_became_peered":"2023-11-28T14:32:09.344782+0000","last_unstale":"2023-11-28T14:33:30.768034+0000","last_undegraded":"2023-11-28T14:33:30.768034+0000","last_fullsized":"2023-11-28T14:33:30.768034+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6302592,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":35,"num_read_kb":22807,"num_write":31,"num_write_kb":9129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"92'55","reported_seq":220,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.149321+0000","last_change":"2023-11-28T14:33:32.504367+0000","last_active":"2023-11-28T14:33:33.149321+0000","last_peered":"2023-11-28T14:33:33.149321+0000","last_clean":"2023-11-28T14:33:33.149321+0000","last_became_active":"2023-11-28T14:32:09.326269+0000","last_became_peered":"2023-11-28T14:32:09.326269+0000","last_unstale":"2023-11-28T14:33:33.149321+0000","last_undegraded":"2023-11-28T14:33:33.149321+0000","last_fullsized":"2023-11-28T14:33:33.149321+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5445435,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":44,"num_read_kb":40985,"num_write":38,"num_write_kb":10508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"90'48","reported_seq":204,"reported_epoch":97,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:33:32.490488+0000","last_change":"2023-11-28T14:33:32.490488+0000","last_active":"2023-11-28T14:33:32.490488+0000","last_peered":"2023-11-28T14:33:32.490488+0000","last_clean":"2023-11-28T14:33:32.490488+0000","last_became_active":"2023-11-28T14:32:09.308846+0000","last_became_peered":"2023-11-28T14:32:09.308846+0000","last_unstale":"2023-11-28T14:33:32.490488+0000","last_undegraded":"2023-11-28T14:33:32.490488+0000","last_fullsized":"2023-11-28T14:33:32.490488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7747994,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":21,"num_read_kb":9822,"num_write":30,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"80'36","reported_seq":204,"reported_epoch":97,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:33:32.492076+0000","last_change":"2023-11-28T14:33:32.492076+0000","last_active":"2023-11-28T14:33:32.492076+0000","last_peered":"2023-11-28T14:33:32.492076+0000","last_clean":"2023-11-28T14:33:32.492076+0000","last_became_active":"2023-11-28T14:32:09.316660+0000","last_became_peered":"2023-11-28T14:32:09.316660+0000","last_unstale":"2023-11-28T14:33:32.492076+0000","last_undegraded":"2023-11-28T14:33:32.492076+0000","last_fullsized":"2023-11-28T14:33:32.492076+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075714,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":16,"num_read_kb":13969,"num_write":31,"num_write_kb":8807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"92'53","reported_seq":220,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.166962+0000","last_change":"2023-11-28T14:33:32.518573+0000","last_active":"2023-11-28T14:33:33.166962+0000","last_peered":"2023-11-28T14:33:33.166962+0000","last_clean":"2023-11-28T14:33:33.166962+0000","last_became_active":"2023-11-28T14:32:09.358025+0000","last_became_peered":"2023-11-28T14:32:09.358025+0000","last_unstale":"2023-11-28T14:33:33.166962+0000","last_undegraded":"2023-11-28T14:33:33.166962+0000","last_fullsized":"2023-11-28T14:33:33.166962+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226977,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":27,"num_read_kb":18499,"num_write":33,"num_write_kb":9164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":270,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.153984+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:33:33.153984+0000","last_peered":"2023-11-28T14:33:33.153984+0000","last_clean":"2023-11-28T14:33:33.153984+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:33:33.153984+0000","last_undegraded":"2023-11-28T14:33:33.153984+0000","last_fullsized":"2023-11-28T14:33:33.153984+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":143,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.160111+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:33:33.160111+0000","last_peered":"2023-11-28T14:33:33.160111+0000","last_clean":"2023-11-28T14:33:33.160111+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:33:33.160111+0000","last_undegraded":"2023-11-28T14:33:33.160111+0000","last_fullsized":"2023-11-28T14:33:33.160111+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"97'137","reported_seq":302,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.167538+0000","last_change":"2023-11-28T14:33:32.542414+0000","last_active":"2023-11-28T14:33:33.167538+0000","last_peered":"2023-11-28T14:33:33.167538+0000","last_clean":"2023-11-28T14:33:33.167538+0000","last_became_active":"2023-11-28T14:32:09.335805+0000","last_became_peered":"2023-11-28T14:32:09.335805+0000","last_unstale":"2023-11-28T14:33:33.167538+0000","last_undegraded":"2023-11-28T14:33:33.167538+0000","last_fullsized":"2023-11-28T14:33:33.167538+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16756513,"num_objects":19,"num_object_clones":11,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":58190,"num_write":93,"num_write_kb":25857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":135,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-11-28T14:33:30.773706+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:33:30.773706+0000","last_peered":"2023-11-28T14:33:30.773706+0000","last_clean":"2023-11-28T14:33:30.773706+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:33:30.773706+0000","last_undegraded":"2023-11-28T14:33:30.773706+0000","last_fullsized":"2023-11-28T14:33:30.773706+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"84'72","reported_seq":216,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.219718+0000","last_change":"2023-11-28T14:33:32.488849+0000","last_active":"2023-11-28T14:33:33.219718+0000","last_peered":"2023-11-28T14:33:33.219718+0000","last_clean":"2023-11-28T14:33:33.219718+0000","last_became_active":"2023-11-28T14:32:09.350412+0000","last_became_peered":"2023-11-28T14:32:09.350412+0000","last_unstale":"2023-11-28T14:33:33.219718+0000","last_undegraded":"2023-11-28T14:33:33.219718+0000","last_fullsized":"2023-11-28T14:33:33.219718+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588278,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":31,"num_read_kb":14625,"num_write":38,"num_write_kb":11086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":143,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.162695+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:33:33.162695+0000","last_peered":"2023-11-28T14:33:33.162695+0000","last_clean":"2023-11-28T14:33:33.162695+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:33:33.162695+0000","last_undegraded":"2023-11-28T14:33:33.162695+0000","last_fullsized":"2023-11-28T14:33:33.162695+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'32","reported_seq":184,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.148297+0000","last_change":"2023-11-28T14:33:32.493761+0000","last_active":"2023-11-28T14:33:33.148297+0000","last_peered":"2023-11-28T14:33:33.148297+0000","last_clean":"2023-11-28T14:33:33.148297+0000","last_became_active":"2023-11-28T14:32:09.343664+0000","last_became_peered":"2023-11-28T14:32:09.343664+0000","last_unstale":"2023-11-28T14:33:33.148297+0000","last_undegraded":"2023-11-28T14:33:33.148297+0000","last_fullsized":"2023-11-28T14:33:33.148297+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932805,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":19,"num_write_kb":5527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":135,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-11-28T14:33:30.771229+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:33:30.771229+0000","last_peered":"2023-11-28T14:33:30.771229+0000","last_clean":"2023-11-28T14:33:30.771229+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:33:30.771229+0000","last_undegraded":"2023-11-28T14:33:30.771229+0000","last_fullsized":"2023-11-28T14:33:30.771229+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'11","reported_seq":178,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-11-28T14:33:32.165517+0000","last_change":"2023-11-28T14:33:32.188413+0000","last_active":"2023-11-28T14:33:32.165517+0000","last_peered":"2023-11-28T14:33:32.165517+0000","last_clean":"2023-11-28T14:33:32.165517+0000","last_became_active":"2023-11-28T14:32:09.311653+0000","last_became_peered":"2023-11-28T14:32:09.311653+0000","last_unstale":"2023-11-28T14:33:32.165517+0000","last_undegraded":"2023-11-28T14:33:32.165517+0000","last_fullsized":"2023-11-28T14:33:32.165517+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2100875,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10016,"num_write":8,"num_write_kb":1910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"98'41","reported_seq":208,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.326328+0000","last_change":"2023-11-28T14:33:32.537602+0000","last_active":"2023-11-28T14:33:33.326328+0000","last_peered":"2023-11-28T14:33:33.326328+0000","last_clean":"2023-11-28T14:33:33.326328+0000","last_became_active":"2023-11-28T14:32:09.316944+0000","last_became_peered":"2023-11-28T14:32:09.316944+0000","last_unstale":"2023-11-28T14:33:33.326328+0000","last_undegraded":"2023-11-28T14:33:33.326328+0000","last_fullsized":"2023-11-28T14:33:33.326328+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3008453,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":19,"num_read_kb":4787,"num_write":30,"num_write_kb":8891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":137,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-11-28T14:33:32.165298+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:33:32.165298+0000","last_peered":"2023-11-28T14:33:32.165298+0000","last_clean":"2023-11-28T14:33:32.165298+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:33:32.165298+0000","last_undegraded":"2023-11-28T14:33:32.165298+0000","last_fullsized":"2023-11-28T14:33:32.165298+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"97'35","reported_seq":197,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-11-28T14:33:32.158506+0000","last_change":"2023-11-28T14:33:32.194413+0000","last_active":"2023-11-28T14:33:32.158506+0000","last_peered":"2023-11-28T14:33:32.158506+0000","last_clean":"2023-11-28T14:33:32.158506+0000","last_became_active":"2023-11-28T14:32:09.309510+0000","last_became_peered":"2023-11-28T14:32:09.309510+0000","last_unstale":"2023-11-28T14:33:32.158506+0000","last_undegraded":"2023-11-28T14:33:32.158506+0000","last_fullsized":"2023-11-28T14:33:32.158506+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2974839,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":10725,"num_write":25,"num_write_kb":7305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":137,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.157331+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:33:33.157331+0000","last_peered":"2023-11-28T14:33:33.157331+0000","last_clean":"2023-11-28T14:33:33.157331+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:33:33.157331+0000","last_undegraded":"2023-11-28T14:33:33.157331+0000","last_fullsized":"2023-11-28T14:33:33.157331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"98'46","reported_seq":214,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.347751+0000","last_change":"2023-11-28T14:33:32.504719+0000","last_active":"2023-11-28T14:33:33.347751+0000","last_peered":"2023-11-28T14:33:33.347751+0000","last_clean":"2023-11-28T14:33:33.347751+0000","last_became_active":"2023-11-28T14:32:09.356991+0000","last_became_peered":"2023-11-28T14:32:09.356991+0000","last_unstale":"2023-11-28T14:33:33.347751+0000","last_undegraded":"2023-11-28T14:33:33.347751+0000","last_fullsized":"2023-11-28T14:33:33.347751+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7529536,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":24,"num_read_kb":13849,"num_write":39,"num_write_kb":11589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":137,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-11-28T14:33:32.168878+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:33:32.168878+0000","last_peered":"2023-11-28T14:33:32.168878+0000","last_clean":"2023-11-28T14:33:32.168878+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:33:32.168878+0000","last_undegraded":"2023-11-28T14:33:32.168878+0000","last_fullsized":"2023-11-28T14:33:32.168878+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"97'46","reported_seq":198,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-11-28T14:33:32.159265+0000","last_change":"2023-11-28T14:33:32.194939+0000","last_active":"2023-11-28T14:33:32.159265+0000","last_peered":"2023-11-28T14:33:32.159265+0000","last_clean":"2023-11-28T14:33:32.159265+0000","last_became_active":"2023-11-28T14:32:09.337548+0000","last_became_peered":"2023-11-28T14:32:09.337548+0000","last_unstale":"2023-11-28T14:33:32.159265+0000","last_undegraded":"2023-11-28T14:33:32.159265+0000","last_fullsized":"2023-11-28T14:33:32.159265+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857190,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":21,"num_read_kb":9891,"num_write":26,"num_write_kb":7070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":143,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.161073+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:33:33.161073+0000","last_peered":"2023-11-28T14:33:33.161073+0000","last_clean":"2023-11-28T14:33:33.161073+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:33:33.161073+0000","last_undegraded":"2023-11-28T14:33:33.161073+0000","last_fullsized":"2023-11-28T14:33:33.161073+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94471916,"num_objects":135,"num_object_clones":82,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":484,"num_read_kb":311948,"num_write":603,"num_write_kb":170978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":893,"ondisk_log_size":893,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":90672,"kb_used_data":59688,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94281168,"statfs":{"total":96636764160,"available":96543916032,"internally_reserved":0,"allocated":61120512,"data_stored":66367868,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100020,"kb_used_data":69036,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94271820,"statfs":{"total":96636764160,"available":96534343680,"internally_reserved":0,"allocated":70692864,"data_stored":77693553,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67696,"kb_used_data":40872,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94304144,"statfs":{"total":96636764160,"available":96567443456,"internally_reserved":0,"allocated":41852928,"data_stored":45701778,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87592,"kb_used_data":60768,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94284248,"statfs":{"total":96636764160,"available":96547069952,"internally_reserved":0,"allocated":62226432,"data_stored":67097529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:33:37.521 INFO:tasks.rados.rados.0.smithi055.stdout:363: finishing write tid 2 to smithi05583006-8 2023-11-28T14:33:37.521 INFO:tasks.rados.rados.0.smithi055.stdout:363: finishing write tid 3 to smithi05583006-8 2023-11-28T14:33:37.522 INFO:tasks.rados.rados.0.smithi055.stdout:363: finishing write tid 4 to smithi05583006-8 2023-11-28T14:33:37.522 INFO:tasks.rados.rados.0.smithi055.stdout:363: oid 8 updating version 0 to 148 2023-11-28T14:33:37.522 INFO:tasks.rados.rados.0.smithi055.stdout:363: oid 8 updating version 148 to 150 2023-11-28T14:33:37.522 INFO:tasks.rados.rados.0.smithi055.stdout:363: oid 8 version 150 is already newer than 149 2023-11-28T14:33:37.522 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 8 v 150 (ObjNum 187 snap 47 seq_num 187) dirty exists 2023-11-28T14:33:37.522 INFO:tasks.rados.rados.0.smithi055.stdout:363: left oid 8 (ObjNum 187 snap 47 seq_num 187) 2023-11-28T14:33:37.522 INFO:tasks.rados.rados.0.smithi055.stdout:365: expect (ObjNum 174 snap 40 seq_num 174) 2023-11-28T14:33:37.634 INFO:tasks.rados.rados.0.smithi055.stdout:358: done (6 left) 2023-11-28T14:33:37.634 INFO:tasks.rados.rados.0.smithi055.stdout:361: done (5 left) 2023-11-28T14:33:37.635 INFO:tasks.rados.rados.0.smithi055.stdout:363: done (4 left) 2023-11-28T14:33:37.635 INFO:tasks.rados.rados.0.smithi055.stdout:364: done (3 left) 2023-11-28T14:33:37.635 INFO:tasks.rados.rados.0.smithi055.stdout:365: done (2 left) 2023-11-28T14:33:37.635 INFO:tasks.rados.rados.0.smithi055.stdout:366: done (1 left) 2023-11-28T14:33:37.635 INFO:tasks.rados.rados.0.smithi055.stdout:367: done (0 left) 2023-11-28T14:33:37.635 INFO:tasks.rados.rados.0.smithi055.stdout:368: snap_create 2023-11-28T14:33:38.139 INFO:tasks.rados.rados.0.smithi055.stdout:368: done (0 left) 2023-11-28T14:33:38.139 INFO:tasks.rados.rados.0.smithi055.stdout:369: read oid 13 snap 31 2023-11-28T14:33:38.139 INFO:tasks.rados.rados.0.smithi055.stdout:369: expect deleted 2023-11-28T14:33:38.139 INFO:tasks.rados.rados.0.smithi055.stdout:370: write oid 23 current snap is 48 2023-11-28T14:33:38.139 INFO:tasks.rados.rados.0.smithi055.stdout:370: seq_num 189 ranges {776523=614533,1976979=757720} 2023-11-28T14:33:38.146 INFO:tasks.rados.rados.0.smithi055.stdout:370: writing smithi05583006-23 from 776523 to 1391056 tid 1 2023-11-28T14:33:38.154 INFO:tasks.rados.rados.0.smithi055.stdout:370: writing smithi05583006-23 from 1976979 to 2734699 tid 2 2023-11-28T14:33:38.154 INFO:tasks.rados.rados.0.smithi055.stdout:371: read oid 16 snap 42 2023-11-28T14:33:38.154 INFO:tasks.rados.rados.0.smithi055.stdout:371: expect (ObjNum 72 snap 1 seq_num 72) 2023-11-28T14:33:38.154 INFO:tasks.rados.rados.0.smithi055.stdout:372: write oid 33 current snap is 48 2023-11-28T14:33:38.154 INFO:tasks.rados.rados.0.smithi055.stdout:372: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2023-11-28T14:33:38.162 INFO:tasks.rados.rados.0.smithi055.stdout:372: writing smithi05583006-33 from 793330 to 1483112 tid 1 2023-11-28T14:33:38.170 INFO:tasks.rados.rados.0.smithi055.stdout:372: writing smithi05583006-33 from 2035461 to 2653192 tid 2 2023-11-28T14:33:38.173 INFO:tasks.rados.rados.0.smithi055.stdout:372: writing smithi05583006-33 from 3101545 to 3193330 tid 3 2023-11-28T14:33:38.173 INFO:tasks.rados.rados.0.smithi055.stdout:373: snap_remove snap 38 2023-11-28T14:33:38.517 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:33:38.634 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:38.632+0000 7f4317b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:38.638 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:38.637+0000 7f4317b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:39.153 INFO:tasks.rados.rados.0.smithi055.stdout:373: done (4 left) 2023-11-28T14:33:39.153 INFO:tasks.rados.rados.0.smithi055.stdout:374: delete oid 49 current snap is 48 2023-11-28T14:33:39.153 INFO:tasks.rados.rados.0.smithi055.stdout:370: finishing write tid 1 to smithi05583006-23 2023-11-28T14:33:39.154 INFO:tasks.rados.rados.0.smithi055.stdout:370: finishing write tid 2 to smithi05583006-23 2023-11-28T14:33:39.154 INFO:tasks.rados.rados.0.smithi055.stdout:370: finishing write tid 3 to smithi05583006-23 2023-11-28T14:33:39.154 INFO:tasks.rados.rados.0.smithi055.stdout:370: finishing write tid 4 to smithi05583006-23 2023-11-28T14:33:39.154 INFO:tasks.rados.rados.0.smithi055.stdout:370: oid 23 updating version 0 to 60 2023-11-28T14:33:39.154 INFO:tasks.rados.rados.0.smithi055.stdout:370: oid 23 updating version 60 to 62 2023-11-28T14:33:39.154 INFO:tasks.rados.rados.0.smithi055.stdout:370: oid 23 version 62 is already newer than 62 2023-11-28T14:33:39.154 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 23 v 62 (ObjNum 189 snap 48 seq_num 189) dirty exists 2023-11-28T14:33:39.155 INFO:tasks.rados.rados.0.smithi055.stdout:370: left oid 23 (ObjNum 189 snap 48 seq_num 189) 2023-11-28T14:33:39.220 INFO:tasks.rados.rados.0.smithi055.stdout:370: done (4 left) 2023-11-28T14:33:39.220 INFO:tasks.rados.rados.0.smithi055.stdout:374: done (3 left) 2023-11-28T14:33:39.220 INFO:tasks.rados.rados.0.smithi055.stdout:375: write oid 12 current snap is 48 2023-11-28T14:33:39.220 INFO:tasks.rados.rados.0.smithi055.stdout:375: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2023-11-28T14:33:39.226 INFO:tasks.rados.rados.0.smithi055.stdout:375: writing smithi05583006-12 from 426944 to 983577 tid 1 2023-11-28T14:33:39.231 INFO:tasks.rados.rados.0.smithi055.stdout:375: writing smithi05583006-12 from 1468778 to 2090178 tid 2 2023-11-28T14:33:39.237 INFO:tasks.rados.rados.0.smithi055.stdout:375: writing smithi05583006-12 from 2872744 to 3226944 tid 3 2023-11-28T14:33:39.237 INFO:tasks.rados.rados.0.smithi055.stdout:376: read oid 30 snap -1 2023-11-28T14:33:39.238 INFO:tasks.rados.rados.0.smithi055.stdout:376: expect deleted 2023-11-28T14:33:39.238 INFO:tasks.rados.rados.0.smithi055.stdout:377: snap_remove snap 46 2023-11-28T14:33:40.159 INFO:tasks.rados.rados.0.smithi055.stdout:377: done (5 left) 2023-11-28T14:33:40.159 INFO:tasks.rados.rados.0.smithi055.stdout:378: snap_create 2023-11-28T14:33:40.159 INFO:tasks.rados.rados.0.smithi055.stdout:375: finishing write tid 1 to smithi05583006-12 2023-11-28T14:33:40.159 INFO:tasks.rados.rados.0.smithi055.stdout:375: finishing write tid 2 to smithi05583006-12 2023-11-28T14:33:40.159 INFO:tasks.rados.rados.0.smithi055.stdout:375: finishing write tid 3 to smithi05583006-12 2023-11-28T14:33:40.159 INFO:tasks.rados.rados.0.smithi055.stdout:375: finishing write tid 4 to smithi05583006-12 2023-11-28T14:33:40.159 INFO:tasks.rados.rados.0.smithi055.stdout:375: finishing write tid 5 to smithi05583006-12 2023-11-28T14:33:40.159 INFO:tasks.rados.rados.0.smithi055.stdout:375: oid 12 updating version 0 to 42 2023-11-28T14:33:40.159 INFO:tasks.rados.rados.0.smithi055.stdout:375: oid 12 version 42 is already newer than 41 2023-11-28T14:33:40.159 INFO:tasks.rados.rados.0.smithi055.stdout:375: oid 12 updating version 42 to 43 2023-11-28T14:33:40.160 INFO:tasks.rados.rados.0.smithi055.stdout:375: oid 12 version 43 is already newer than 40 2023-11-28T14:33:40.160 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 12 v 43 (ObjNum 192 snap 48 seq_num 192) dirty exists 2023-11-28T14:33:40.160 INFO:tasks.rados.rados.0.smithi055.stdout:375: left oid 12 (ObjNum 192 snap 48 seq_num 192) 2023-11-28T14:33:41.168 INFO:tasks.rados.rados.0.smithi055.stdout:375: done (5 left) 2023-11-28T14:33:41.168 INFO:tasks.rados.rados.0.smithi055.stdout:378: done (4 left) 2023-11-28T14:33:41.168 INFO:tasks.rados.rados.0.smithi055.stdout:379: write oid 35 current snap is 49 2023-11-28T14:33:41.168 INFO:tasks.rados.rados.0.smithi055.stdout:379: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2023-11-28T14:33:41.174 INFO:tasks.rados.rados.0.smithi055.stdout:379: writing smithi05583006-35 from 443751 to 1075633 tid 1 2023-11-28T14:33:41.182 INFO:tasks.rados.rados.0.smithi055.stdout:379: writing smithi05583006-35 from 1810907 to 2575965 tid 2 2023-11-28T14:33:41.186 INFO:tasks.rados.rados.0.smithi055.stdout:379: writing smithi05583006-35 from 3067461 to 3243751 tid 3 2023-11-28T14:33:41.186 INFO:tasks.rados.rados.0.smithi055.stdout:380: write oid 24 current snap is 49 2023-11-28T14:33:41.186 INFO:tasks.rados.rados.0.smithi055.stdout:380: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2023-11-28T14:33:41.192 INFO:tasks.rados.rados.0.smithi055.stdout:380: writing smithi05583006-24 from 460558 to 1167689 tid 1 2023-11-28T14:33:41.199 INFO:tasks.rados.rados.0.smithi055.stdout:380: writing smithi05583006-24 from 1869389 to 2378105 tid 2 2023-11-28T14:33:41.203 INFO:tasks.rados.rados.0.smithi055.stdout:380: writing smithi05583006-24 from 3094884 to 3260558 tid 3 2023-11-28T14:33:41.203 INFO:tasks.rados.rados.0.smithi055.stdout:381: delete oid 22 current snap is 49 2023-11-28T14:33:41.276 INFO:tasks.rados.rados.0.smithi055.stdout:380: finishing write tid 1 to smithi05583006-24 2023-11-28T14:33:41.290 INFO:tasks.rados.rados.0.smithi055.stdout:379: finishing write tid 1 to smithi05583006-35 2023-11-28T14:33:41.299 INFO:tasks.rados.rados.0.smithi055.stdout:380: finishing write tid 2 to smithi05583006-24 2023-11-28T14:33:41.316 INFO:tasks.rados.rados.0.smithi055.stdout:379: finishing write tid 2 to smithi05583006-35 2023-11-28T14:33:41.318 INFO:tasks.rados.rados.0.smithi055.stdout:380: finishing write tid 3 to smithi05583006-24 2023-11-28T14:33:41.333 INFO:tasks.rados.rados.0.smithi055.stdout:379: finishing write tid 3 to smithi05583006-35 2023-11-28T14:33:41.334 INFO:tasks.rados.rados.0.smithi055.stdout:380: finishing write tid 4 to smithi05583006-24 2023-11-28T14:33:41.337 INFO:tasks.rados.rados.0.smithi055.stdout:379: finishing write tid 4 to smithi05583006-35 2023-11-28T14:33:41.338 INFO:tasks.rados.rados.0.smithi055.stdout:380: finishing write tid 5 to smithi05583006-24 2023-11-28T14:33:41.338 INFO:tasks.rados.rados.0.smithi055.stdout:380: oid 24 updating version 0 to 47 2023-11-28T14:33:41.338 INFO:tasks.rados.rados.0.smithi055.stdout:380: oid 24 updating version 47 to 48 2023-11-28T14:33:41.338 INFO:tasks.rados.rados.0.smithi055.stdout:380: oid 24 version 48 is already newer than 45 2023-11-28T14:33:41.338 INFO:tasks.rados.rados.0.smithi055.stdout:380: oid 24 version 48 is already newer than 46 2023-11-28T14:33:41.338 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 24 v 48 (ObjNum 194 snap 49 seq_num 194) dirty exists 2023-11-28T14:33:41.338 INFO:tasks.rados.rados.0.smithi055.stdout:380: left oid 24 (ObjNum 194 snap 49 seq_num 194) 2023-11-28T14:33:41.339 INFO:tasks.rados.rados.0.smithi055.stdout:379: finishing write tid 5 to smithi05583006-35 2023-11-28T14:33:41.339 INFO:tasks.rados.rados.0.smithi055.stdout:379: oid 35 updating version 0 to 59 2023-11-28T14:33:41.339 INFO:tasks.rados.rados.0.smithi055.stdout:379: oid 35 version 59 is already newer than 57 2023-11-28T14:33:41.339 INFO:tasks.rados.rados.0.smithi055.stdout:379: oid 35 version 59 is already newer than 59 2023-11-28T14:33:41.339 INFO:tasks.rados.rados.0.smithi055.stdout:379: oid 35 version 59 is already newer than 56 2023-11-28T14:33:41.340 INFO:tasks.rados.rados.0.smithi055.stdout:update_object_version oid 35 v 59 (ObjNum 193 snap 49 seq_num 193) dirty exists 2023-11-28T14:33:41.340 INFO:tasks.rados.rados.0.smithi055.stdout:379: left oid 35 (ObjNum 193 snap 49 seq_num 193) 2023-11-28T14:33:42.448 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:33:42.448 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:33:42.449 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:33:42.575 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:42.571+0000 7faf1a39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:42.655 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:42.577+0000 7faf1a39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:42.765 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:33:42.766 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:33:42.775 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2023-11-28T14:33:42.477737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":89801726,"num_objects":142,"num_object_clones":87,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":610,"num_read_kb":330838,"num_write":754,"num_write_kb":176817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1022,"ondisk_log_size":1022,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":364604,"kb_used_data":248988,"kb_used_omap":0,"kb_used_meta":115584,"kb_avail":377122756,"statfs":{"total":386547056640,"available":386173702144,"internally_reserved":0,"allocated":254963712,"data_stored":280886273,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":118358016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":18796,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.341623"},"pg_stats":[{"pgid":"3.f","version":"102'163","reported_seq":317,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-11-28T14:33:37.147911+0000","last_change":"2023-11-28T14:33:37.240883+0000","last_active":"2023-11-28T14:33:37.147911+0000","last_peered":"2023-11-28T14:33:37.147911+0000","last_clean":"2023-11-28T14:33:37.147911+0000","last_became_active":"2023-11-28T14:32:09.321992+0000","last_became_peered":"2023-11-28T14:32:09.321992+0000","last_unstale":"2023-11-28T14:33:37.147911+0000","last_undegraded":"2023-11-28T14:33:37.147911+0000","last_fullsized":"2023-11-28T14:33:37.147911+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11227043,"num_objects":23,"num_object_clones":15,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":74,"num_read_kb":47888,"num_write":112,"num_write_kb":32509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'26","reported_seq":203,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-28T14:33:38.485167+0000","last_change":"2023-11-28T14:33:37.212866+0000","last_active":"2023-11-28T14:33:38.485167+0000","last_peered":"2023-11-28T14:33:38.485167+0000","last_clean":"2023-11-28T14:33:38.485167+0000","last_became_active":"2023-11-28T14:32:09.308848+0000","last_became_peered":"2023-11-28T14:32:09.308848+0000","last_unstale":"2023-11-28T14:33:38.485167+0000","last_undegraded":"2023-11-28T14:33:38.485167+0000","last_fullsized":"2023-11-28T14:33:38.485167+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050454,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":17312,"num_write":19,"num_write_kb":5379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'54","reported_seq":223,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.165072+0000","last_change":"2023-11-28T14:33:32.516199+0000","last_active":"2023-11-28T14:33:33.165072+0000","last_peered":"2023-11-28T14:33:33.165072+0000","last_clean":"2023-11-28T14:33:33.165072+0000","last_became_active":"2023-11-28T14:32:09.337560+0000","last_became_peered":"2023-11-28T14:32:09.337560+0000","last_unstale":"2023-11-28T14:33:33.165072+0000","last_undegraded":"2023-11-28T14:33:33.165072+0000","last_fullsized":"2023-11-28T14:33:33.165072+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428694,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":24018,"num_write":36,"num_write_kb":9320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"92'51","reported_seq":226,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-11-28T14:33:37.184454+0000","last_change":"2023-11-28T14:33:37.184454+0000","last_active":"2023-11-28T14:33:37.184454+0000","last_peered":"2023-11-28T14:33:37.184454+0000","last_clean":"2023-11-28T14:33:37.184454+0000","last_became_active":"2023-11-28T14:32:09.344782+0000","last_became_peered":"2023-11-28T14:32:09.344782+0000","last_unstale":"2023-11-28T14:33:37.184454+0000","last_undegraded":"2023-11-28T14:33:37.184454+0000","last_fullsized":"2023-11-28T14:33:37.184454+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2840383,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":36,"num_read_kb":22807,"num_write":31,"num_write_kb":9129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"92'55","reported_seq":230,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-28T14:33:38.154143+0000","last_change":"2023-11-28T14:33:37.277555+0000","last_active":"2023-11-28T14:33:38.154143+0000","last_peered":"2023-11-28T14:33:38.154143+0000","last_clean":"2023-11-28T14:33:38.154143+0000","last_became_active":"2023-11-28T14:32:09.326269+0000","last_became_peered":"2023-11-28T14:32:09.326269+0000","last_unstale":"2023-11-28T14:33:38.154143+0000","last_undegraded":"2023-11-28T14:33:38.154143+0000","last_fullsized":"2023-11-28T14:33:38.154143+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5445435,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":44,"num_read_kb":40985,"num_write":38,"num_write_kb":10508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"102'61","reported_seq":226,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-11-28T14:33:37.163555+0000","last_change":"2023-11-28T14:33:37.245524+0000","last_active":"2023-11-28T14:33:37.163555+0000","last_peered":"2023-11-28T14:33:37.163555+0000","last_clean":"2023-11-28T14:33:37.163555+0000","last_became_active":"2023-11-28T14:32:09.308846+0000","last_became_peered":"2023-11-28T14:32:09.308846+0000","last_unstale":"2023-11-28T14:33:37.163555+0000","last_undegraded":"2023-11-28T14:33:37.163555+0000","last_fullsized":"2023-11-28T14:33:37.163555+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8168169,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":24,"num_read_kb":9824,"num_write":39,"num_write_kb":10872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"80'36","reported_seq":217,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-11-28T14:33:37.142878+0000","last_change":"2023-11-28T14:33:37.152647+0000","last_active":"2023-11-28T14:33:37.142878+0000","last_peered":"2023-11-28T14:33:37.142878+0000","last_clean":"2023-11-28T14:33:37.142878+0000","last_became_active":"2023-11-28T14:32:09.316660+0000","last_became_peered":"2023-11-28T14:32:09.316660+0000","last_unstale":"2023-11-28T14:33:37.142878+0000","last_undegraded":"2023-11-28T14:33:37.142878+0000","last_fullsized":"2023-11-28T14:33:37.142878+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075714,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":16,"num_read_kb":13969,"num_write":31,"num_write_kb":8807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"92'53","reported_seq":220,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.166962+0000","last_change":"2023-11-28T14:33:32.518573+0000","last_active":"2023-11-28T14:33:33.166962+0000","last_peered":"2023-11-28T14:33:33.166962+0000","last_clean":"2023-11-28T14:33:33.166962+0000","last_became_active":"2023-11-28T14:32:09.358025+0000","last_became_peered":"2023-11-28T14:32:09.358025+0000","last_unstale":"2023-11-28T14:33:33.166962+0000","last_undegraded":"2023-11-28T14:33:33.166962+0000","last_fullsized":"2023-11-28T14:33:33.166962+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226977,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":27,"num_read_kb":18499,"num_write":33,"num_write_kb":9164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":278,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-28T14:33:38.155735+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:33:38.155735+0000","last_peered":"2023-11-28T14:33:38.155735+0000","last_clean":"2023-11-28T14:33:38.155735+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:33:38.155735+0000","last_undegraded":"2023-11-28T14:33:38.155735+0000","last_fullsized":"2023-11-28T14:33:38.155735+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":143,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.160111+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:33:33.160111+0000","last_peered":"2023-11-28T14:33:33.160111+0000","last_clean":"2023-11-28T14:33:33.160111+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:33:33.160111+0000","last_undegraded":"2023-11-28T14:33:33.160111+0000","last_fullsized":"2023-11-28T14:33:33.160111+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"97'137","reported_seq":302,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.167538+0000","last_change":"2023-11-28T14:33:32.542414+0000","last_active":"2023-11-28T14:33:33.167538+0000","last_peered":"2023-11-28T14:33:33.167538+0000","last_clean":"2023-11-28T14:33:33.167538+0000","last_became_active":"2023-11-28T14:32:09.335805+0000","last_became_peered":"2023-11-28T14:32:09.335805+0000","last_unstale":"2023-11-28T14:33:33.167538+0000","last_undegraded":"2023-11-28T14:33:33.167538+0000","last_fullsized":"2023-11-28T14:33:33.167538+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16756513,"num_objects":19,"num_object_clones":11,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":58190,"num_write":93,"num_write_kb":25857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":147,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-11-28T14:33:37.143556+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:33:37.143556+0000","last_peered":"2023-11-28T14:33:37.143556+0000","last_clean":"2023-11-28T14:33:37.143556+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:33:37.143556+0000","last_undegraded":"2023-11-28T14:33:37.143556+0000","last_fullsized":"2023-11-28T14:33:37.143556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"84'72","reported_seq":226,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-28T14:33:38.146625+0000","last_change":"2023-11-28T14:33:37.236049+0000","last_active":"2023-11-28T14:33:38.146625+0000","last_peered":"2023-11-28T14:33:38.146625+0000","last_clean":"2023-11-28T14:33:38.146625+0000","last_became_active":"2023-11-28T14:32:09.350412+0000","last_became_peered":"2023-11-28T14:32:09.350412+0000","last_unstale":"2023-11-28T14:33:38.146625+0000","last_undegraded":"2023-11-28T14:33:38.146625+0000","last_fullsized":"2023-11-28T14:33:38.146625+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588278,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":31,"num_read_kb":14625,"num_write":38,"num_write_kb":11086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":143,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.162695+0000","last_change":"2023-11-28T14:32:02.742394+0000","last_active":"2023-11-28T14:33:33.162695+0000","last_peered":"2023-11-28T14:33:33.162695+0000","last_clean":"2023-11-28T14:33:33.162695+0000","last_became_active":"2023-11-28T14:32:02.738252+0000","last_became_peered":"2023-11-28T14:32:02.738252+0000","last_unstale":"2023-11-28T14:33:33.162695+0000","last_undegraded":"2023-11-28T14:33:33.162695+0000","last_fullsized":"2023-11-28T14:33:33.162695+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'32","reported_seq":194,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-28T14:33:38.148674+0000","last_change":"2023-11-28T14:33:37.222063+0000","last_active":"2023-11-28T14:33:38.148674+0000","last_peered":"2023-11-28T14:33:38.148674+0000","last_clean":"2023-11-28T14:33:38.148674+0000","last_became_active":"2023-11-28T14:32:09.343664+0000","last_became_peered":"2023-11-28T14:32:09.343664+0000","last_unstale":"2023-11-28T14:33:38.148674+0000","last_undegraded":"2023-11-28T14:33:38.148674+0000","last_fullsized":"2023-11-28T14:33:38.148674+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932805,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":19,"num_write_kb":5527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":147,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-11-28T14:33:37.144499+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:33:37.144499+0000","last_peered":"2023-11-28T14:33:37.144499+0000","last_clean":"2023-11-28T14:33:37.144499+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:33:37.144499+0000","last_undegraded":"2023-11-28T14:33:37.144499+0000","last_fullsized":"2023-11-28T14:33:37.144499+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'11","reported_seq":188,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-28T14:33:38.489630+0000","last_change":"2023-11-28T14:33:37.210722+0000","last_active":"2023-11-28T14:33:38.489630+0000","last_peered":"2023-11-28T14:33:38.489630+0000","last_clean":"2023-11-28T14:33:38.489630+0000","last_became_active":"2023-11-28T14:32:09.311653+0000","last_became_peered":"2023-11-28T14:32:09.311653+0000","last_unstale":"2023-11-28T14:33:38.489630+0000","last_undegraded":"2023-11-28T14:33:38.489630+0000","last_fullsized":"2023-11-28T14:33:38.489630+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2100875,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10016,"num_write":8,"num_write_kb":1910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"98'41","reported_seq":208,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.326328+0000","last_change":"2023-11-28T14:33:32.537602+0000","last_active":"2023-11-28T14:33:33.326328+0000","last_peered":"2023-11-28T14:33:33.326328+0000","last_clean":"2023-11-28T14:33:33.326328+0000","last_became_active":"2023-11-28T14:32:09.316944+0000","last_became_peered":"2023-11-28T14:32:09.316944+0000","last_unstale":"2023-11-28T14:33:33.326328+0000","last_undegraded":"2023-11-28T14:33:33.326328+0000","last_fullsized":"2023-11-28T14:33:33.326328+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3008453,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":19,"num_read_kb":4787,"num_write":30,"num_write_kb":8891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":145,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-28T14:33:38.486731+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:33:38.486731+0000","last_peered":"2023-11-28T14:33:38.486731+0000","last_clean":"2023-11-28T14:33:38.486731+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:33:38.486731+0000","last_undegraded":"2023-11-28T14:33:38.486731+0000","last_fullsized":"2023-11-28T14:33:38.486731+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"101'37","reported_seq":207,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-28T14:33:38.485886+0000","last_change":"2023-11-28T14:33:37.212955+0000","last_active":"2023-11-28T14:33:38.485886+0000","last_peered":"2023-11-28T14:33:38.485886+0000","last_clean":"2023-11-28T14:33:38.485886+0000","last_became_active":"2023-11-28T14:32:09.309510+0000","last_became_peered":"2023-11-28T14:32:09.309510+0000","last_unstale":"2023-11-28T14:33:38.485886+0000","last_undegraded":"2023-11-28T14:33:38.485886+0000","last_fullsized":"2023-11-28T14:33:38.485886+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2974839,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":10725,"num_write":25,"num_write_kb":7305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":145,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-28T14:33:38.150677+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:33:38.150677+0000","last_peered":"2023-11-28T14:33:38.150677+0000","last_clean":"2023-11-28T14:33:38.150677+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:33:38.150677+0000","last_undegraded":"2023-11-28T14:33:38.150677+0000","last_fullsized":"2023-11-28T14:33:38.150677+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"98'46","reported_seq":227,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-28T14:33:38.157419+0000","last_change":"2023-11-28T14:33:37.292852+0000","last_active":"2023-11-28T14:33:38.157419+0000","last_peered":"2023-11-28T14:33:38.157419+0000","last_clean":"2023-11-28T14:33:38.157419+0000","last_became_active":"2023-11-28T14:32:09.356991+0000","last_became_peered":"2023-11-28T14:32:09.356991+0000","last_unstale":"2023-11-28T14:33:38.157419+0000","last_undegraded":"2023-11-28T14:33:38.157419+0000","last_fullsized":"2023-11-28T14:33:38.157419+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7529536,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":30,"num_read_kb":19216,"num_write":39,"num_write_kb":11589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":145,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-28T14:33:38.486105+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:33:38.486105+0000","last_peered":"2023-11-28T14:33:38.486105+0000","last_clean":"2023-11-28T14:33:38.486105+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:33:38.486105+0000","last_undegraded":"2023-11-28T14:33:38.486105+0000","last_fullsized":"2023-11-28T14:33:38.486105+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"101'50","reported_seq":212,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-28T14:33:38.479590+0000","last_change":"2023-11-28T14:33:37.213967+0000","last_active":"2023-11-28T14:33:38.479590+0000","last_peered":"2023-11-28T14:33:38.479590+0000","last_clean":"2023-11-28T14:33:38.479590+0000","last_became_active":"2023-11-28T14:32:09.337548+0000","last_became_peered":"2023-11-28T14:32:09.337548+0000","last_unstale":"2023-11-28T14:33:38.479590+0000","last_undegraded":"2023-11-28T14:33:38.479590+0000","last_fullsized":"2023-11-28T14:33:38.479590+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857190,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":17882,"num_write":26,"num_write_kb":7070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":143,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-11-28T14:33:33.161073+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:33:33.161073+0000","last_peered":"2023-11-28T14:33:33.161073+0000","last_clean":"2023-11-28T14:33:33.161073+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:33:33.161073+0000","last_undegraded":"2023-11-28T14:33:33.161073+0000","last_fullsized":"2023-11-28T14:33:33.161073+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":89211358,"num_objects":140,"num_object_clones":87,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":507,"num_read_kb":330747,"num_write":617,"num_write_kb":174923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":925,"ondisk_log_size":925,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":97112,"kb_used_data":66128,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94274728,"statfs":{"total":96636764160,"available":96537321472,"internally_reserved":0,"allocated":67715072,"data_stored":75055506,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":107748,"kb_used_data":76764,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94264092,"statfs":{"total":96636764160,"available":96526430208,"internally_reserved":0,"allocated":78606336,"data_stored":87650705,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72152,"kb_used_data":45328,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94299688,"statfs":{"total":96636764160,"available":96562880512,"internally_reserved":0,"allocated":46415872,"data_stored":51082533,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87592,"kb_used_data":60768,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94284248,"statfs":{"total":96636764160,"available":96547069952,"internally_reserved":0,"allocated":62226432,"data_stored":67097529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:33:44.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-11-28T14:33:47.777 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:33:47.778 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:33:47.778 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-11-28T14:33:47.778 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-11-28T14:33:47.905 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:47.900+0000 7f380106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:47.908 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:47.907+0000 7f380106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:47.921 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:33:47.921 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:33:47.921 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:33:47.921 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:33:47.921 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:33:47.921 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:33:47.922 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:33:47.922 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:33:47.922 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:33:47.922 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:33:47.922 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:33:47.922 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:33:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:33:47.924 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:33:47.925 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-11-28T14:33:48.038 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:48.032+0000 7fc0558f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:48.041 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:33:48.040+0000 7fc0558f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:50.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~6s 2023-11-28T14:33:55.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2023-11-28T14:34:01.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~17s 2023-11-28T14:34:06.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~22s 2023-11-28T14:34:09.174 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: osd down 2023-11-28T14:34:09.175 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.1 2023-11-28T14:34:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-28T14:34:09.179 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:34:09.180 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-11-28T14:34:09.309 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:34:09.304+0000 7f31bcf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:34:09.312 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:34:09.311+0000 7f31bcf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:34:09.325 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:34:09.325 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:34:09.325 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:34:09.325 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:34:09.325 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:34:09.326 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:34:09.326 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:34:09.326 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:34:09.326 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:34:09.326 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:34:09.326 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:34:09.326 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:34:09.328 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:34:09.329 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:34:09.329 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-11-28T14:34:09.445 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:34:09.440+0000 7f8c672ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:34:09.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:34:09.447+0000 7f8c672ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:34:09.461 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:34:09.461 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:34:09.461 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:34:09.461 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:34:09.461 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:34:09.461 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:34:09.461 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:34:09.461 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:34:09.461 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:34:09.462 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:34:09.462 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:34:09.462 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:34:09.464 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:34:09.464 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:34:11.974 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-11-28T14:34:11.974 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~28s 2023-11-28T14:34:11.974 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-11-28T14:34:14.465 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:34:14.465 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:34:14.465 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-11-28T14:34:14.466 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-11-28T14:34:14.592 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:34:14.586+0000 7f9468617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:34:14.597 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:34:14.595+0000 7f9468617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:34:17.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2023-11-28T14:34:17.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~33s 2023-11-28T14:34:17.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2023-11-28T14:34:22.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2023-11-28T14:34:22.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~38s 2023-11-28T14:34:22.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~11s 2023-11-28T14:34:27.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2023-11-28T14:34:27.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~43s 2023-11-28T14:34:27.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~16s 2023-11-28T14:34:33.188 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~21s 2023-11-28T14:34:33.188 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~49s 2023-11-28T14:34:33.188 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~21s 2023-11-28T14:34:38.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~27s 2023-11-28T14:34:38.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~54s 2023-11-28T14:34:38.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~27s 2023-11-28T14:34:43.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~32s 2023-11-28T14:34:43.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~59s 2023-11-28T14:34:43.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~32s 2023-11-28T14:34:49.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~37s 2023-11-28T14:34:49.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s 2023-11-28T14:34:49.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~37s 2023-11-28T14:34:54.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~42s 2023-11-28T14:34:54.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~70s 2023-11-28T14:34:54.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~42s 2023-11-28T14:34:59.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~48s 2023-11-28T14:34:59.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~75s 2023-11-28T14:34:59.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~48s 2023-11-28T14:35:05.008 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~53s 2023-11-28T14:35:05.009 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~81s 2023-11-28T14:35:05.009 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~53s 2023-11-28T14:35:10.150 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:35:10.147+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:35:13.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~58s 2023-11-28T14:35:13.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~86s 2023-11-28T14:35:13.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~58s 2023-11-28T14:35:15.150 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:35:15.149+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:35:19.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~67s 2023-11-28T14:35:23.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~95s 2023-11-28T14:35:23.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~67s 2023-11-28T14:35:23.427 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:35:20.151+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:35:25.153 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:35:25.152+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:35:28.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~77s 2023-11-28T14:35:28.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~104s 2023-11-28T14:35:28.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~77s 2023-11-28T14:35:30.154 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:35:30.153+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:35:34.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~82s 2023-11-28T14:35:34.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~110s 2023-11-28T14:35:34.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~82s 2023-11-28T14:35:35.155 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:35:35.154+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:35:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:35:38.556 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T14:35:39.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~87s 2023-11-28T14:35:39.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~115s 2023-11-28T14:35:39.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~87s 2023-11-28T14:35:40.156 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:35:40.155+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:35:40.558 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:35:40.683 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:35:40.680+0000 7f0d17a0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:35:40.686 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:35:40.685+0000 7f0d17a0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:35:44.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~93s 2023-11-28T14:35:44.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~120s 2023-11-28T14:35:44.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~93s 2023-11-28T14:35:45.157 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:35:45.156+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:35:49.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~98s 2023-11-28T14:35:49.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~125s 2023-11-28T14:35:49.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~98s 2023-11-28T14:35:50.158 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:35:50.157+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:35:55.159 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:35:55.157+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:35:55.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~103s 2023-11-28T14:35:55.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~131s 2023-11-28T14:35:55.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~103s 2023-11-28T14:36:00.161 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:36:00.159+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:36:00.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~109s 2023-11-28T14:36:00.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~136s 2023-11-28T14:36:00.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~109s 2023-11-28T14:36:05.162 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:36:05.161+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:36:05.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~114s 2023-11-28T14:36:05.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~141s 2023-11-28T14:36:05.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~114s 2023-11-28T14:36:10.163 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:36:10.162+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:36:11.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~119s 2023-11-28T14:36:11.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~147s 2023-11-28T14:36:11.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~119s 2023-11-28T14:36:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:36:14.509 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:36:14.509 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-11-28T14:36:14.635 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:36:14.629+0000 7f592062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:36:14.638 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:36:14.637+0000 7f592062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:36:14.641 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.1 2023-11-28T14:36:14.645 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-28T14:36:14.645 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:36:14.645 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-11-28T14:36:14.758 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:36:14.753+0000 7f063cf11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:36:14.761 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:36:14.761+0000 7f063cf11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:36:15.164 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:36:15.163+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:36:16.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~124s 2023-11-28T14:36:16.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~152s 2023-11-28T14:36:16.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~124s 2023-11-28T14:36:20.166 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:36:20.165+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:36:21.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~130s 2023-11-28T14:36:21.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~157s 2023-11-28T14:36:21.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~130s 2023-11-28T14:36:25.167 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:36:25.166+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:36:27.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~135s 2023-11-28T14:36:27.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~163s 2023-11-28T14:36:27.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~135s 2023-11-28T14:36:30.168 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:36:30.167+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:36:32.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~140s 2023-11-28T14:36:32.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~168s 2023-11-28T14:36:32.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~140s 2023-11-28T14:36:35.169 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:36:35.168+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:36:37.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~146s 2023-11-28T14:36:37.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~173s 2023-11-28T14:36:37.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~146s 2023-11-28T14:36:40.172 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:36:40.170+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:36:42.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~151s 2023-11-28T14:36:42.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~179s 2023-11-28T14:36:42.982 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~151s 2023-11-28T14:36:45.173 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:36:45.171+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:36:48.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~156s 2023-11-28T14:36:48.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~184s 2023-11-28T14:36:48.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~156s 2023-11-28T14:36:50.174 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:36:50.172+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:36:53.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~162s 2023-11-28T14:36:53.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~189s 2023-11-28T14:36:53.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~162s 2023-11-28T14:36:55.175 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:36:55.174+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:36:58.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~167s 2023-11-28T14:36:58.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~194s 2023-11-28T14:36:58.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~167s 2023-11-28T14:37:00.177 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:37:00.175+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:37:04.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~172s 2023-11-28T14:37:04.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~200s 2023-11-28T14:37:04.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~172s 2023-11-28T14:37:05.178 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:37:05.177+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:37:09.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~178s 2023-11-28T14:37:09.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~205s 2023-11-28T14:37:09.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~178s 2023-11-28T14:37:10.179 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:37:10.177+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:37:14.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~183s 2023-11-28T14:37:14.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~210s 2023-11-28T14:37:14.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~183s 2023-11-28T14:37:15.179 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:37:15.178+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:37:20.104 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~188s 2023-11-28T14:37:20.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~216s 2023-11-28T14:37:20.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~188s 2023-11-28T14:37:20.182 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:37:20.180+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:37:25.183 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:37:25.182+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:37:25.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~193s 2023-11-28T14:37:25.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~221s 2023-11-28T14:37:25.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~193s 2023-11-28T14:37:30.184 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:37:30.182+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:37:30.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~199s 2023-11-28T14:37:30.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~226s 2023-11-28T14:37:30.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~199s 2023-11-28T14:37:35.184 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:37:35.182+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:37:36.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~204s 2023-11-28T14:37:36.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~232s 2023-11-28T14:37:36.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~204s 2023-11-28T14:37:40.186 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:37:40.184+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:37:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:37:40.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T14:37:41.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~209s 2023-11-28T14:37:45.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~237s 2023-11-28T14:37:45.494 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~209s 2023-11-28T14:37:45.494 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:37:45.495 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:37:45.186+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:37:45.618 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:37:45.616+0000 7f3af8c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:37:45.621 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:37:45.621+0000 7f3af8c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:37:50.189 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:37:50.187+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:37:50.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~219s 2023-11-28T14:37:50.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~246s 2023-11-28T14:37:50.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~219s 2023-11-28T14:37:55.190 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:37:55.188+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:37:56.100 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~224s 2023-11-28T14:37:56.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~252s 2023-11-28T14:37:56.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~224s 2023-11-28T14:38:00.191 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:38:00.190+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:38:01.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~229s 2023-11-28T14:38:01.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~257s 2023-11-28T14:38:01.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~229s 2023-11-28T14:38:05.192 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:38:05.191+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:38:06.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~235s 2023-11-28T14:38:06.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~262s 2023-11-28T14:38:06.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~235s 2023-11-28T14:38:10.193 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:38:10.192+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:38:12.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~240s 2023-11-28T14:38:12.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~268s 2023-11-28T14:38:12.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~240s 2023-11-28T14:38:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:38:14.682 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:38:14.682 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-11-28T14:38:14.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:38:14.803+0000 7f1edd19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:38:14.811 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:38:14.810+0000 7f1edd19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:38:14.826 INFO:teuthology.orchestra.run.smithi055.stderr:no valid command found; 10 closest matches: 2023-11-28T14:38:14.826 INFO:teuthology.orchestra.run.smithi055.stderr:config get 2023-11-28T14:38:14.826 INFO:teuthology.orchestra.run.smithi055.stderr:config help 2023-11-28T14:38:14.826 INFO:teuthology.orchestra.run.smithi055.stderr:config set ... 2023-11-28T14:38:14.826 INFO:teuthology.orchestra.run.smithi055.stderr:config show 2023-11-28T14:38:14.826 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_ops 2023-11-28T14:38:14.826 INFO:teuthology.orchestra.run.smithi055.stderr:dump_historic_slow_ops 2023-11-28T14:38:14.826 INFO:teuthology.orchestra.run.smithi055.stderr:dump_metrics [] 2023-11-28T14:38:14.826 INFO:teuthology.orchestra.run.smithi055.stderr:dump_ops_in_flight 2023-11-28T14:38:14.826 INFO:teuthology.orchestra.run.smithi055.stderr:dump_pgstate_history 2023-11-28T14:38:14.827 INFO:teuthology.orchestra.run.smithi055.stderr:dump_recovery_reservations 2023-11-28T14:38:14.827 INFO:teuthology.orchestra.run.smithi055.stderr:Error EINVAL: invalid command 2023-11-28T14:38:14.829 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-28T14:38:14.829 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:38:15.194 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:38:15.193+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:38:17.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~245s 2023-11-28T14:38:17.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~273s 2023-11-28T14:38:17.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~245s 2023-11-28T14:38:19.830 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:38:19.830 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:38:19.831 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:38:19.957 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:38:19.952+0000 7f7bce79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:38:19.960 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:38:19.958+0000 7f7bce79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:38:20.143 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-28T14:38:20.144 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-11-28T14:38:20.156 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2023-11-28T14:38:18.545912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":78893983,"num_objects":121,"num_object_clones":74,"num_object_copies":363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":564,"num_read_kb":298191,"num_write":670,"num_write_kb":153828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1064,"ondisk_log_size":1064,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":378168,"kb_used_data":262552,"kb_used_omap":0,"kb_used_meta":115584,"kb_avail":377109192,"statfs":{"total":386547056640,"available":386159812608,"internally_reserved":0,"allocated":268853248,"data_stored":294717108,"data_compressed":90147,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":118358016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003328"},"pg_stats":[{"pgid":"3.f","version":"104'165","reported_seq":324,"reported_epoch":106,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:33:41.224075+0000","last_change":"2023-11-28T14:33:41.224075+0000","last_active":"2023-11-28T14:33:41.224075+0000","last_peered":"2023-11-28T14:33:41.224075+0000","last_clean":"2023-11-28T14:33:41.224075+0000","last_became_active":"2023-11-28T14:32:09.321992+0000","last_became_peered":"2023-11-28T14:32:09.321992+0000","last_unstale":"2023-11-28T14:33:41.224075+0000","last_undegraded":"2023-11-28T14:33:41.224075+0000","last_fullsized":"2023-11-28T14:33:41.224075+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11227043,"num_objects":23,"num_object_clones":15,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":74,"num_read_kb":47888,"num_write":112,"num_write_kb":32509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'26","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-28T14:33:41.179671+0000","last_change":"2023-11-28T14:33:40.179727+0000","last_active":"2023-11-28T14:33:41.179671+0000","last_peered":"2023-11-28T14:33:41.179671+0000","last_clean":"2023-11-28T14:33:41.179671+0000","last_became_active":"2023-11-28T14:32:09.308848+0000","last_became_peered":"2023-11-28T14:32:09.308848+0000","last_unstale":"2023-11-28T14:33:41.179671+0000","last_undegraded":"2023-11-28T14:33:41.179671+0000","last_fullsized":"2023-11-28T14:33:41.179671+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050454,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":17312,"num_write":19,"num_write_kb":5379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'54","reported_seq":9,"reported_epoch":108,"state":"active+undersized","last_fresh":"2023-11-28T14:34:10.185194+0000","last_change":"2023-11-28T14:34:10.184757+0000","last_active":"2023-11-28T14:34:10.185194+0000","last_peered":"2023-11-28T14:34:10.185194+0000","last_clean":"2023-11-28T14:32:08.119105+0000","last_became_active":"2023-11-28T14:34:10.184757+0000","last_became_peered":"2023-11-28T14:34:10.184757+0000","last_unstale":"2023-11-28T14:34:10.185194+0000","last_undegraded":"2023-11-28T14:34:10.185194+0000","last_fullsized":"2023-11-28T14:34:10.145194+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":38,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":9800,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"106'57","reported_seq":234,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-28T14:33:41.225980+0000","last_change":"2023-11-28T14:33:41.291502+0000","last_active":"2023-11-28T14:33:41.225980+0000","last_peered":"2023-11-28T14:33:41.225980+0000","last_clean":"2023-11-28T14:33:41.225980+0000","last_became_active":"2023-11-28T14:32:09.344782+0000","last_became_peered":"2023-11-28T14:32:09.344782+0000","last_unstale":"2023-11-28T14:33:41.225980+0000","last_undegraded":"2023-11-28T14:33:41.225980+0000","last_fullsized":"2023-11-28T14:33:41.225980+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2840383,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":36,"num_read_kb":22807,"num_write":31,"num_write_kb":9129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"106'59","reported_seq":245,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-28T14:34:10.151714+0000","last_change":"2023-11-28T14:33:41.258204+0000","last_active":"2023-11-28T14:34:10.151714+0000","last_peered":"2023-11-28T14:34:10.151714+0000","last_clean":"2023-11-28T14:34:10.151714+0000","last_became_active":"2023-11-28T14:32:09.326269+0000","last_became_peered":"2023-11-28T14:32:09.326269+0000","last_unstale":"2023-11-28T14:34:10.151714+0000","last_undegraded":"2023-11-28T14:34:10.151714+0000","last_fullsized":"2023-11-28T14:34:10.151714+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8689186,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":45,"num_read_kb":40986,"num_write":43,"num_write_kb":12047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"104'63","reported_seq":232,"reported_epoch":106,"state":"active+clean+snaptrim","last_fresh":"2023-11-28T14:33:41.226141+0000","last_change":"2023-11-28T14:33:39.172333+0000","last_active":"2023-11-28T14:33:41.226141+0000","last_peered":"2023-11-28T14:33:41.226141+0000","last_clean":"2023-11-28T14:33:41.226141+0000","last_became_active":"2023-11-28T14:32:09.308846+0000","last_became_peered":"2023-11-28T14:32:09.308846+0000","last_unstale":"2023-11-28T14:33:41.226141+0000","last_undegraded":"2023-11-28T14:33:41.226141+0000","last_fullsized":"2023-11-28T14:33:41.226141+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8168169,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":24,"num_read_kb":9824,"num_write":39,"num_write_kb":10872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"80'36","reported_seq":225,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-28T14:33:41.219853+0000","last_change":"2023-11-28T14:33:41.229401+0000","last_active":"2023-11-28T14:33:41.219853+0000","last_peered":"2023-11-28T14:33:41.219853+0000","last_clean":"2023-11-28T14:33:41.219853+0000","last_became_active":"2023-11-28T14:32:09.316660+0000","last_became_peered":"2023-11-28T14:32:09.316660+0000","last_unstale":"2023-11-28T14:33:41.219853+0000","last_undegraded":"2023-11-28T14:33:41.219853+0000","last_fullsized":"2023-11-28T14:33:41.219853+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075714,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":16,"num_read_kb":13969,"num_write":31,"num_write_kb":8807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"103'62","reported_seq":4,"reported_epoch":108,"state":"peering","last_fresh":"2023-11-28T14:34:09.213679+0000","last_change":"2023-11-28T14:34:09.213679+0000","last_active":"2023-11-28T14:32:08.119105+0000","last_peered":"2023-11-28T14:32:08.119105+0000","last_clean":"2023-11-28T14:32:08.119105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T14:34:09.213679+0000","last_undegraded":"2023-11-28T14:34:09.213679+0000","last_fullsized":"2023-11-28T14:34:09.213679+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":4854,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":286,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-28T14:34:10.152516+0000","last_change":"2023-11-28T14:31:59.619422+0000","last_active":"2023-11-28T14:34:10.152516+0000","last_peered":"2023-11-28T14:34:10.152516+0000","last_clean":"2023-11-28T14:34:10.152516+0000","last_became_active":"2023-11-28T14:31:59.614566+0000","last_became_peered":"2023-11-28T14:31:59.614566+0000","last_unstale":"2023-11-28T14:34:10.152516+0000","last_undegraded":"2023-11-28T14:34:10.152516+0000","last_fullsized":"2023-11-28T14:34:10.152516+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:31:58.565483+0000","last_clean_scrub_stamp":"2023-11-28T14:31:58.565483+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":143,"reported_epoch":98,"state":"stale+active+clean","last_fresh":"2023-11-28T14:33:33.160111+0000","last_change":"2023-11-28T14:32:02.759211+0000","last_active":"2023-11-28T14:33:33.160111+0000","last_peered":"2023-11-28T14:33:33.160111+0000","last_clean":"2023-11-28T14:33:33.160111+0000","last_became_active":"2023-11-28T14:32:02.755817+0000","last_became_peered":"2023-11-28T14:32:02.755817+0000","last_unstale":"2023-11-28T14:34:09.170202+0000","last_undegraded":"2023-11-28T14:33:33.160111+0000","last_fullsized":"2023-11-28T14:33:33.160111+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"97'137","reported_seq":302,"reported_epoch":98,"state":"stale+active+clean","last_fresh":"2023-11-28T14:33:33.167538+0000","last_change":"2023-11-28T14:33:32.542414+0000","last_active":"2023-11-28T14:33:33.167538+0000","last_peered":"2023-11-28T14:33:33.167538+0000","last_clean":"2023-11-28T14:33:33.167538+0000","last_became_active":"2023-11-28T14:32:09.335805+0000","last_became_peered":"2023-11-28T14:32:09.335805+0000","last_unstale":"2023-11-28T14:34:09.170204+0000","last_undegraded":"2023-11-28T14:33:33.167538+0000","last_fullsized":"2023-11-28T14:33:33.167538+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16756513,"num_objects":19,"num_object_clones":11,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":58190,"num_write":93,"num_write_kb":25857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":153,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-28T14:33:41.225100+0000","last_change":"2023-11-28T14:32:02.771128+0000","last_active":"2023-11-28T14:33:41.225100+0000","last_peered":"2023-11-28T14:33:41.225100+0000","last_clean":"2023-11-28T14:33:41.225100+0000","last_became_active":"2023-11-28T14:32:02.766260+0000","last_became_peered":"2023-11-28T14:32:02.766260+0000","last_unstale":"2023-11-28T14:33:41.225100+0000","last_undegraded":"2023-11-28T14:33:41.225100+0000","last_fullsized":"2023-11-28T14:33:41.225100+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"84'72","reported_seq":239,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-28T14:34:10.196730+0000","last_change":"2023-11-28T14:34:10.196319+0000","last_active":"2023-11-28T14:34:10.196730+0000","last_peered":"2023-11-28T14:34:10.196730+0000","last_clean":"2023-11-28T14:33:41.231504+0000","last_became_active":"2023-11-28T14:34:10.196319+0000","last_became_peered":"2023-11-28T14:34:10.196319+0000","last_unstale":"2023-11-28T14:34:10.196730+0000","last_undegraded":"2023-11-28T14:34:10.150691+0000","last_fullsized":"2023-11-28T14:34:10.149902+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":38,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588278,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":31,"num_read_kb":14625,"num_write":38,"num_write_kb":11086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":4,"reported_epoch":108,"state":"peering","last_fresh":"2023-11-28T14:34:09.215412+0000","last_change":"2023-11-28T14:34:09.215412+0000","last_active":"2023-11-28T14:32:01.622522+0000","last_peered":"2023-11-28T14:32:01.622522+0000","last_clean":"2023-11-28T14:32:01.622522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T14:34:09.215412+0000","last_undegraded":"2023-11-28T14:34:09.215412+0000","last_fullsized":"2023-11-28T14:34:09.215412+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"55'32","reported_seq":202,"reported_epoch":108,"state":"peering","last_fresh":"2023-11-28T14:34:09.200788+0000","last_change":"2023-11-28T14:34:09.200788+0000","last_active":"2023-11-28T14:34:09.200788+0000","last_peered":"2023-11-28T14:33:41.224249+0000","last_clean":"2023-11-28T14:33:41.224249+0000","last_became_active":"2023-11-28T14:32:09.343664+0000","last_became_peered":"2023-11-28T14:32:09.343664+0000","last_unstale":"2023-11-28T14:34:09.200788+0000","last_undegraded":"2023-11-28T14:34:09.200788+0000","last_fullsized":"2023-11-28T14:34:09.200788+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932805,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":19,"num_write_kb":5527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":153,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-28T14:33:41.225603+0000","last_change":"2023-11-28T14:32:02.757496+0000","last_active":"2023-11-28T14:33:41.225603+0000","last_peered":"2023-11-28T14:33:41.225603+0000","last_clean":"2023-11-28T14:33:41.225603+0000","last_became_active":"2023-11-28T14:32:02.752193+0000","last_became_peered":"2023-11-28T14:32:02.752193+0000","last_unstale":"2023-11-28T14:33:41.225603+0000","last_undegraded":"2023-11-28T14:33:41.225603+0000","last_fullsized":"2023-11-28T14:33:41.225603+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'11","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-28T14:33:41.191956+0000","last_change":"2023-11-28T14:33:40.183422+0000","last_active":"2023-11-28T14:33:41.191956+0000","last_peered":"2023-11-28T14:33:41.191956+0000","last_clean":"2023-11-28T14:33:41.191956+0000","last_became_active":"2023-11-28T14:32:09.311653+0000","last_became_peered":"2023-11-28T14:32:09.311653+0000","last_unstale":"2023-11-28T14:33:41.191956+0000","last_undegraded":"2023-11-28T14:33:41.191956+0000","last_fullsized":"2023-11-28T14:33:41.191956+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2100875,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10016,"num_write":8,"num_write_kb":1910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"101'43","reported_seq":4,"reported_epoch":108,"state":"peering","last_fresh":"2023-11-28T14:34:09.213297+0000","last_change":"2023-11-28T14:34:09.213297+0000","last_active":"2023-11-28T14:32:08.119105+0000","last_peered":"2023-11-28T14:32:08.119105+0000","last_clean":"2023-11-28T14:32:08.119105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T14:34:09.213297+0000","last_undegraded":"2023-11-28T14:34:09.213297+0000","last_fullsized":"2023-11-28T14:34:09.213297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":151,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-28T14:33:41.178310+0000","last_change":"2023-11-28T14:32:02.774468+0000","last_active":"2023-11-28T14:33:41.178310+0000","last_peered":"2023-11-28T14:33:41.178310+0000","last_clean":"2023-11-28T14:33:41.178310+0000","last_became_active":"2023-11-28T14:32:02.769144+0000","last_became_peered":"2023-11-28T14:32:02.769144+0000","last_unstale":"2023-11-28T14:33:41.178310+0000","last_undegraded":"2023-11-28T14:33:41.178310+0000","last_fullsized":"2023-11-28T14:33:41.178310+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"106'48","reported_seq":225,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-28T14:33:41.319012+0000","last_change":"2023-11-28T14:33:40.179768+0000","last_active":"2023-11-28T14:33:41.319012+0000","last_peered":"2023-11-28T14:33:41.319012+0000","last_clean":"2023-11-28T14:33:41.319012+0000","last_became_active":"2023-11-28T14:32:09.309510+0000","last_became_peered":"2023-11-28T14:32:09.309510+0000","last_unstale":"2023-11-28T14:33:41.319012+0000","last_undegraded":"2023-11-28T14:33:41.319012+0000","last_fullsized":"2023-11-28T14:33:41.319012+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6487502,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":10727,"num_write":35,"num_write_kb":10152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":153,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-28T14:34:10.148161+0000","last_change":"2023-11-28T14:32:02.735174+0000","last_active":"2023-11-28T14:34:10.148161+0000","last_peered":"2023-11-28T14:34:10.148161+0000","last_clean":"2023-11-28T14:34:10.148161+0000","last_became_active":"2023-11-28T14:32:02.730843+0000","last_became_peered":"2023-11-28T14:32:02.730843+0000","last_unstale":"2023-11-28T14:34:10.148161+0000","last_undegraded":"2023-11-28T14:34:10.148161+0000","last_fullsized":"2023-11-28T14:34:10.148161+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"106'50","reported_seq":279,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-28T14:34:10.304457+0000","last_change":"2023-11-28T14:34:10.306717+0000","last_active":"2023-11-28T14:34:10.304457+0000","last_peered":"2023-11-28T14:34:10.304457+0000","last_clean":"2023-11-28T14:33:41.224346+0000","last_became_active":"2023-11-28T14:34:10.188886+0000","last_became_peered":"2023-11-28T14:34:10.188886+0000","last_unstale":"2023-11-28T14:34:10.304457+0000","last_undegraded":"2023-11-28T14:34:10.156556+0000","last_fullsized":"2023-11-28T14:34:10.155736+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5529503,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":30,"num_read_kb":19216,"num_write":39,"num_write_kb":11589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":151,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-28T14:33:41.178914+0000","last_change":"2023-11-28T14:32:02.751927+0000","last_active":"2023-11-28T14:33:41.178914+0000","last_peered":"2023-11-28T14:33:41.178914+0000","last_clean":"2023-11-28T14:33:41.178914+0000","last_became_active":"2023-11-28T14:32:02.748743+0000","last_became_peered":"2023-11-28T14:32:02.748743+0000","last_unstale":"2023-11-28T14:33:41.178914+0000","last_undegraded":"2023-11-28T14:33:41.178914+0000","last_fullsized":"2023-11-28T14:33:41.178914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"104'52","reported_seq":218,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-28T14:33:41.177755+0000","last_change":"2023-11-28T14:33:41.178000+0000","last_active":"2023-11-28T14:33:41.177755+0000","last_peered":"2023-11-28T14:33:41.177755+0000","last_clean":"2023-11-28T14:33:41.177755+0000","last_became_active":"2023-11-28T14:32:09.337548+0000","last_became_peered":"2023-11-28T14:32:09.337548+0000","last_unstale":"2023-11-28T14:33:41.177755+0000","last_undegraded":"2023-11-28T14:33:41.177755+0000","last_fullsized":"2023-11-28T14:33:41.177755+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:08.119105+0000","last_clean_scrub_stamp":"2023-11-28T14:32:08.119105+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857190,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":17882,"num_write":26,"num_write_kb":7070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":143,"reported_epoch":98,"state":"stale+active+clean","last_fresh":"2023-11-28T14:33:33.161073+0000","last_change":"2023-11-28T14:32:02.776311+0000","last_active":"2023-11-28T14:33:33.161073+0000","last_peered":"2023-11-28T14:33:33.161073+0000","last_clean":"2023-11-28T14:33:33.161073+0000","last_became_active":"2023-11-28T14:32:02.768439+0000","last_became_peered":"2023-11-28T14:32:02.768439+0000","last_unstale":"2023-11-28T14:34:09.170208+0000","last_undegraded":"2023-11-28T14:33:33.161073+0000","last_fullsized":"2023-11-28T14:33:33.161073+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:32:01.622522+0000","last_clean_scrub_stamp":"2023-11-28T14:32:01.622522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":78303615,"num_objects":119,"num_object_clones":74,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":461,"num_read_kb":298100,"num_write":533,"num_write_kb":151934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":967,"ondisk_log_size":967,"up":42,"acting":42,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":23,"acting":23,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705793,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100396,"kb_used_data":69412,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94271444,"statfs":{"total":96636764160,"available":96533958656,"internally_reserved":0,"allocated":71077888,"data_stored":77820194,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706039,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":112248,"kb_used_data":81264,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94259592,"statfs":{"total":96636764160,"available":96521822208,"internally_reserved":0,"allocated":83214336,"data_stored":92309452,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77932,"kb_used_data":51108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94293908,"statfs":{"total":96636764160,"available":96556961792,"internally_reserved":0,"allocated":52334592,"data_stored":57489933,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87592,"kb_used_data":60768,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94284248,"statfs":{"total":96636764160,"available":96547069952,"internally_reserved":0,"allocated":62226432,"data_stored":67097529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:38:20.157 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-recovery 3.1 3.7 2023-11-28T14:38:20.196 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:38:20.194+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:38:20.271 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:38:20.264+0000 7fcfac55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:38:20.274 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:38:20.272+0000 7fcfac55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:38:20.458 INFO:teuthology.orchestra.run.smithi055.stderr:instructing pg(s) [3.1,3.7] on osd.3 to force-recovery; 2023-11-28T14:38:20.458 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-28T14:38:22.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~251s 2023-11-28T14:38:22.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~278s 2023-11-28T14:38:22.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~251s 2023-11-28T14:38:25.197 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:38:25.195+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:38:25.471 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:38:25.472 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:38:25.472 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-11-28T14:38:25.472 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-11-28T14:38:25.600 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:38:25.593+0000 7fcba946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:38:25.603 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:38:25.601+0000 7fcba946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:38:27.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~256s 2023-11-28T14:38:27.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~283s 2023-11-28T14:38:27.920 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~256s 2023-11-28T14:38:30.198 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:38:30.196+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:38:33.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~261s 2023-11-28T14:38:33.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~289s 2023-11-28T14:38:33.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~261s 2023-11-28T14:38:35.199 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:38:35.198+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:38:38.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~267s 2023-11-28T14:38:38.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~294s 2023-11-28T14:38:38.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~267s 2023-11-28T14:38:40.201 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:38:40.200+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:38:43.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~272s 2023-11-28T14:38:43.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~299s 2023-11-28T14:38:43.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~272s 2023-11-28T14:38:45.202 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:38:45.201+0000 7ff9453f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.55:6800/1538959351 for 29sec e108 v108) 2023-11-28T14:38:49.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~277s 2023-11-28T14:38:49.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~305s 2023-11-28T14:38:49.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~277s 2023-11-28T14:38:49.132 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-11-28T14:38:49.434 INFO:tasks.ceph.osd.3:Sent signal 15 2023-11-28T14:38:49.435 INFO:tasks.ceph.mon.a:Sent signal 15 2023-11-28T14:38:49.435 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-11-28T14:38:49.436 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:38:49.435+0000 7ff9483fc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 81326) UID: 0 2023-11-28T14:38:49.436 INFO:tasks.ceph.mon.a.smithi055.stderr:2023-11-28T14:38:49.435+0000 7ff9483fc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-28T14:38:49.437 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:38:49.436+0000 7f5f41ff7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 80678) UID: 0 2023-11-28T14:38:49.437 INFO:tasks.ceph.mgr.x.smithi060.stderr:2023-11-28T14:38:49.436+0000 7f5f41ff7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-11-28T14:39:45.537 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:39:45.538 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T14:39:47.539 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:39:54.817 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:39:54.815+0000 7f4f8335e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:40:25.515 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:40:25.515 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:40:25.516 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-11-28T14:40:25.644 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:40:25.639+0000 7f2f00d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:41:54.736 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:41:55.236 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T14:41:57.238 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:41:57.362 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:41:57.359+0000 7f4d68260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:42:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:42:25.559 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:42:25.559 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-11-28T14:42:25.684 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:42:25.679+0000 7ff15146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:43:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:43:57.282 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T14:43:59.283 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:43:59.408 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:43:59.405+0000 7fb3c379e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:44:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:44:25.601 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:44:25.602 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-11-28T14:44:25.727 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:44:25.721+0000 7fe82b99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:45:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:45:59.326 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T14:46:01.327 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:46:01.451 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:46:01.449+0000 7fc60295e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:46:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:46:25.643 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:46:30.645 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:46:30.645 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:46:30.645 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-11-28T14:46:30.645 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-11-28T14:46:30.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:46:30.766+0000 7fcc6186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:48:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:48:01.370 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T14:48:03.371 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:48:03.495 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:48:03.493+0000 7f02ddaf7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:48:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:48:30.688 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:48:30.689 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-11-28T14:48:30.815 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:48:30.809+0000 7f57f9e06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:50:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:50:03.415 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T14:50:05.416 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:50:05.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:50:05.538+0000 7fbf7b515640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:50:30.730 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:50:30.731 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:50:30.731 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-11-28T14:50:30.857 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:50:30.851+0000 7f6f7af5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:52:05.459 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:52:05.459 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T14:52:07.460 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:52:07.585 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:52:07.582+0000 7fdf03cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:52:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:52:30.773 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:52:30.774 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-11-28T14:52:30.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:52:30.894+0000 7fc48ab9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:54:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:54:07.504 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T14:54:09.506 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:54:09.631 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:54:09.628+0000 7fbd4f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:54:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:54:30.816 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:54:35.817 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T14:54:35.817 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T14:54:35.818 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-11-28T14:54:35.818 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-11-28T14:54:35.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:54:35.939+0000 7faa0506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:56:09.549 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:56:09.550 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T14:56:11.551 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T14:56:11.674 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:56:11.672+0000 7f85c8006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:56:35.859 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:56:35.860 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:56:35.860 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-11-28T14:56:35.986 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:56:35.980+0000 7fc04995e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:58:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:58:11.595 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T14:58:13.596 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T14:58:13.719 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:58:13.717+0000 7ff815f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:58:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T14:58:35.903 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T14:58:35.904 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-11-28T14:58:36.030 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T14:58:36.024+0000 7ffbaa82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:00:13.638 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:00:13.639 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:00:15.639 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:00:15.763 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:00:15.761+0000 7f865046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:00:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:00:35.946 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T15:00:35.947 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-11-28T15:00:36.072 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:00:36.067+0000 7f109639e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:02:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:02:15.683 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:02:17.684 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:02:17.809 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:02:17.807+0000 7f80c8206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:02:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:02:35.989 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-28T15:02:40.990 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-11-28T15:02:40.990 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-11-28T15:02:40.991 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T15:02:41.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:02:41.113+0000 7f22ed66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:04:17.728 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:04:17.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:04:19.729 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:04:19.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:04:19.851+0000 7fd29891e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:04:41.033 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:04:41.035 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 1417, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 779, in force_cancel_recovery self.force_recovery() File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 748, in force_recovery j = self.ceph_manager.get_pgids_to_force(backfill) File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 2385, in get_pgids_to_force j = self.get_pg_stats(); File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi055 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-28T15:04:41.035 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 1285, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 1417, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 779, in force_cancel_recovery self.force_recovery() File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 748, in force_recovery j = self.ceph_manager.get_pgids_to_force(backfill) File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 2385, in get_pgids_to_force j = self.get_pg_stats(); File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi055 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-28T15:06:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:06:19.772 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:06:21.773 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:06:21.899 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:06:21.897+0000 7f658839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:08:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:08:21.817 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:08:23.819 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:08:23.942 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:08:23.940+0000 7f224e71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:10:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:10:23.862 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:10:25.863 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:10:25.988 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:10:25.985+0000 7fe16306d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:12:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:12:25.907 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:12:27.908 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:12:28.032 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:12:28.029+0000 7fcc4b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:14:27.951 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:14:27.952 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:14:29.953 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:14:30.076 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:14:30.073+0000 7fe36d91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:16:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:16:29.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:16:31.996 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:16:32.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:16:32.116+0000 7f0df5ede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:18:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:18:32.040 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:18:34.041 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:18:34.164 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:18:34.162+0000 7f633e99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:20:34.083 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:20:34.084 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:20:36.085 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:20:36.209 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:20:36.206+0000 7f10fd39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:22:36.127 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:22:36.128 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:22:38.129 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:22:38.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:22:38.250+0000 7f554e59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:24:38.171 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:24:38.172 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:24:40.173 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:24:40.297 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:24:40.294+0000 7fc10906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:26:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:26:40.217 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:26:42.218 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:26:42.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:26:42.339+0000 7f0429a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:28:42.260 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:28:42.261 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:28:44.262 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:28:44.386 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:28:44.383+0000 7f7201e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:30:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:30:44.305 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:30:46.307 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:30:46.430 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:30:46.427+0000 7f91b406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:32:46.349 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:32:46.349 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:32:48.351 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:32:48.474 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:32:48.471+0000 7f520baad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:34:48.393 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:34:48.394 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:34:50.395 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:34:50.520 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:34:50.517+0000 7fab30e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:36:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:36:50.438 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:36:52.440 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:36:52.563 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:36:52.560+0000 7f763a39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:38:52.482 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:38:52.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:38:54.484 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:38:54.608 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:38:54.605+0000 7f13f1115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:40:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:40:54.528 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:40:56.529 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:40:56.653 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:40:56.650+0000 7fe57a06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:42:56.572 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:42:56.573 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:42:58.574 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:42:58.698 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:42:58.695+0000 7f1b4271e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:44:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:44:58.617 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:45:00.618 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:45:00.742 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:45:00.739+0000 7fbfc959e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:47:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:47:00.662 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:47:02.663 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:47:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:47:02.783+0000 7f2b2cb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:49:02.705 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:49:02.706 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:49:04.708 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:49:04.832 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:49:04.829+0000 7f3e9f15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:51:04.750 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:51:04.751 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:51:06.752 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:51:06.875 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:51:06.873+0000 7f998f95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:53:06.794 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:53:06.795 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:53:08.797 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:53:08.920 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:53:08.917+0000 7fb9f575e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:55:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:55:08.877 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:55:10.878 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:55:11.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:55:11.000+0000 7fccb162d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:57:10.920 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:57:10.921 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:57:12.922 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T15:57:13.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:57:13.044+0000 7f705a31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T15:59:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T15:59:12.966 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T15:59:14.967 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T15:59:15.091 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T15:59:15.088+0000 7f540a15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:01:15.010 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:01:15.011 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:01:17.012 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:01:17.137 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:01:17.134+0000 7f86dad1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:03:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:03:17.056 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:03:19.058 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:03:19.181 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:03:19.179+0000 7f549a62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:05:19.101 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:05:19.102 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:05:21.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:05:21.228 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:05:21.226+0000 7ff98ba2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:07:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:07:21.146 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:07:23.147 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:07:23.270 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:07:23.267+0000 7fb0ed4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:09:23.189 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:09:23.190 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:09:25.192 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:09:25.315 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:09:25.313+0000 7f03536ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:11:25.233 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:11:25.234 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:11:27.236 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:11:27.361 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:11:27.358+0000 7f02e879e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:13:27.278 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:13:27.278 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:13:29.279 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:13:29.404 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:13:29.402+0000 7f791cd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:15:29.322 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:15:29.323 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:15:31.323 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:15:31.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:15:31.444+0000 7fa600e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:17:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:17:31.368 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:17:33.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:17:33.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:17:33.490+0000 7f2aae39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:19:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:19:33.412 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:19:35.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:19:35.538 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:19:35.534+0000 7fe950d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:21:35.457 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:21:35.457 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:21:37.459 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:21:37.582 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:21:37.580+0000 7f716a42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:23:37.502 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:23:37.503 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:23:39.504 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:23:40.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:23:40.805+0000 7fee46017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:25:40.727 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:25:40.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:25:42.729 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:25:42.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:25:42.850+0000 7f68b0f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:27:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:27:42.772 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:27:44.774 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:27:44.899 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:27:44.897+0000 7feb9c79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:29:44.817 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:29:44.818 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:29:46.819 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:29:46.943 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:29:46.941+0000 7f4a8b39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:31:46.862 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:31:46.863 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:31:48.866 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:31:48.992 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:31:48.990+0000 7fa9ff46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:33:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:33:48.912 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:33:50.914 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:33:51.038 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:33:51.036+0000 7f1bc779e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:35:50.957 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:35:50.958 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:35:52.959 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:35:53.083 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:35:53.081+0000 7f6402d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:37:53.001 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:37:53.002 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:37:55.004 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:37:55.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:37:55.125+0000 7fcc78a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:39:55.046 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:39:55.047 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:39:57.048 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:39:57.172 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:39:57.169+0000 7f5fe0c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:41:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:41:57.092 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:41:59.093 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:41:59.216 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:41:59.214+0000 7f888f22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:43:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:43:59.137 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:44:01.138 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:44:01.262 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:44:01.260+0000 7f840b424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:46:01.181 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:46:01.181 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:46:03.182 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:46:03.306 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:46:03.304+0000 7fd828e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:48:03.225 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:48:03.226 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:48:05.228 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:48:05.351 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:48:05.349+0000 7fea3282d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:50:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:50:05.271 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:50:07.272 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:50:07.397 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:50:07.394+0000 7f874dd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:52:07.316 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:52:07.317 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:52:09.318 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:52:09.442 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:52:09.440+0000 7f496d06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:54:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:54:09.362 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:54:11.364 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:54:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:54:11.486+0000 7f057f59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:56:11.406 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:56:11.407 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:56:13.408 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T16:56:13.533 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:56:13.531+0000 7f96a226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T16:58:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T16:58:13.452 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T16:58:15.453 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T16:58:15.576 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T16:58:15.574+0000 7fb8e0f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:00:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:00:15.496 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:00:17.497 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:00:17.622 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:00:17.619+0000 7f937139e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:02:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:02:17.540 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:02:19.541 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:02:19.665 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:02:19.663+0000 7f20f199e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:04:19.584 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:04:19.585 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:04:21.586 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:04:21.711 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:04:21.706+0000 7fb518264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:06:21.628 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:06:21.629 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:06:23.630 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:06:23.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:06:23.751+0000 7f1cce75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:08:23.672 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:08:23.673 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:08:25.675 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:08:25.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:08:25.798+0000 7f5e254ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:10:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:10:25.722 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:10:27.723 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:10:27.847 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:10:27.844+0000 7f616c99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:12:27.766 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:12:27.767 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:12:29.768 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:12:29.892 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:12:29.890+0000 7f688740f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:14:29.810 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:14:29.811 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:14:31.812 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:14:31.935 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:14:31.932+0000 7f75ec51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:16:31.855 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:16:31.855 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:16:33.857 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:16:33.981 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:16:33.978+0000 7ff18d35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:18:33.899 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:18:33.900 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:18:35.901 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:18:36.026 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:18:36.023+0000 7efda5d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:20:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:20:35.945 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:20:37.947 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:20:38.070 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:20:38.068+0000 7fd9b50ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:22:37.990 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:22:37.991 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:22:39.992 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:22:40.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:22:40.114+0000 7fc5ba46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:24:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:24:40.035 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:24:42.037 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:24:42.162 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:24:42.160+0000 7f8ce6ade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:26:42.080 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:26:42.081 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:26:44.082 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:26:44.206 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:26:44.203+0000 7f72dd59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:28:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:28:44.126 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:28:46.127 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:28:46.251 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:28:46.248+0000 7f3ed9f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:30:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:30:46.171 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:30:48.172 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:30:48.295 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:30:48.292+0000 7ffb48d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:32:48.214 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:32:48.215 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:32:50.216 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:32:50.340 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:32:50.338+0000 7fefe1d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:34:50.260 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:34:50.261 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:34:52.262 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:34:52.386 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:34:52.383+0000 7fb380115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:36:52.304 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:36:52.305 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:36:54.307 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:36:54.430 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:36:54.428+0000 7f8cc7d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:38:54.349 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:38:54.350 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:38:56.352 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:38:56.475 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:38:56.473+0000 7f966a95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:40:56.394 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:40:56.395 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:40:58.396 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:40:58.520 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:40:58.518+0000 7f5e178ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:42:58.439 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:42:58.440 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:43:00.441 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:43:00.565 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:43:00.563+0000 7f43dff5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:45:00.483 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:45:00.484 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:45:02.485 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:45:02.609 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:45:02.607+0000 7f432dd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:47:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:47:02.528 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:47:04.529 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:47:04.653 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:47:04.651+0000 7ff97560f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:49:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:49:04.572 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:49:06.574 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:49:06.699 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:49:06.697+0000 7f201f59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:51:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:51:06.617 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:51:08.618 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:51:08.741 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:51:08.739+0000 7fc13cb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:53:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:53:08.662 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:53:10.663 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:53:10.787 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:53:10.784+0000 7f7881cde640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:55:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:55:10.706 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:55:12.706 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:55:12.830 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:55:12.828+0000 7f19dc91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:57:12.749 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:57:12.749 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:57:14.751 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T17:57:14.874 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:57:14.872+0000 7f705c55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T17:59:14.793 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:59:14.794 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T17:59:16.795 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T17:59:16.918 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T17:59:16.916+0000 7f846dcde640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:01:16.837 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:01:16.838 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:01:18.839 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:01:18.963 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:01:18.961+0000 7f85a8a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:03:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:03:18.883 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:03:20.885 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:03:21.009 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:03:21.006+0000 7fa96346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:05:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:05:20.928 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:05:22.929 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:05:23.053 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:05:23.051+0000 7fc463f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:07:22.971 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:07:22.972 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:07:24.973 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:07:25.098 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:07:25.096+0000 7fde1ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:09:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:09:25.020 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:09:27.021 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:09:27.145 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:09:27.142+0000 7f8199ef0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:11:27.063 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:11:27.064 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:11:29.065 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:11:29.190 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:11:29.188+0000 7f60cbd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:13:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:13:29.110 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:13:31.111 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:13:31.235 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:13:31.232+0000 7f3c7cb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:15:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:15:31.155 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:15:33.156 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:15:33.280 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:15:33.277+0000 7f7c7a99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:17:33.199 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:17:33.199 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:17:35.200 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:17:35.324 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:17:35.322+0000 7f977211e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:19:35.243 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:19:35.244 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:19:37.245 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:19:37.368 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:19:37.366+0000 7f314b115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:21:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:21:37.288 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:21:39.290 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:21:39.414 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:21:39.412+0000 7f9cc8c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:23:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:23:39.333 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:23:41.335 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:23:41.458 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:23:41.456+0000 7f46ccd00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:25:41.377 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:25:41.382 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:25:43.383 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:25:47.871 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:25:47.869+0000 7f386940f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:27:47.790 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:27:47.791 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:27:49.792 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:27:49.916 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:27:49.912+0000 7fa212cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:29:49.835 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:29:49.836 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:29:51.837 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:29:51.961 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:29:51.958+0000 7f26994de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:31:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:31:51.880 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:31:53.881 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:31:54.005 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:31:54.002+0000 7fa7f839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:33:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:33:53.925 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:33:55.926 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:33:56.051 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:33:56.048+0000 7f2ceda6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:35:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:35:55.970 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:35:57.971 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:35:58.096 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:35:58.092+0000 7f308615e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:37:58.014 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:37:58.015 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:38:00.017 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:38:00.143 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:38:00.140+0000 7fd41135e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:40:00.060 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:40:00.061 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:40:02.062 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:40:02.187 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:40:02.184+0000 7fc6018de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:42:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:42:02.108 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:42:04.109 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:42:04.234 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:42:04.231+0000 7f1e0b2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:44:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:44:04.153 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:44:06.155 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:44:06.278 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:44:06.274+0000 7f9a6fd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:46:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:46:06.198 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:46:08.199 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:46:08.322 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:46:08.320+0000 7fbe1759e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:48:08.242 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:48:08.243 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:48:10.244 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:48:10.368 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:48:10.365+0000 7f45f90ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:50:10.288 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:50:10.288 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:50:12.290 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:50:12.413 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:50:12.411+0000 7f524866d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:52:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:52:12.334 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:52:14.335 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:52:14.461 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:52:14.458+0000 7f4fbbb55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:54:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:54:14.396 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:54:16.397 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:54:16.521 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:54:16.518+0000 7f4a0c0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:56:16.440 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:56:16.441 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:56:18.442 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T18:56:18.566 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:56:18.562+0000 7f548e711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T18:58:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T18:58:18.485 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T18:58:20.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T18:58:20.610 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T18:58:20.607+0000 7f190e22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:00:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:00:20.530 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:00:22.531 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:00:22.656 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:00:22.652+0000 7ff37a91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:02:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:02:22.575 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:02:24.575 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:02:24.699 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:02:24.696+0000 7f6594f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:04:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:04:24.619 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:04:26.620 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:04:26.744 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:04:26.741+0000 7f829a013640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:06:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:06:26.665 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:06:28.666 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:06:28.791 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:06:28.787+0000 7fc4a9b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:08:28.708 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:08:28.709 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:08:30.710 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:08:30.834 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:08:30.832+0000 7ff7b515e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:10:30.754 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:10:30.756 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:10:32.757 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:10:32.881 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:10:32.878+0000 7f064911e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:12:32.800 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:12:32.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:12:34.802 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:12:34.927 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:12:34.924+0000 7f89a326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:14:34.846 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:14:34.847 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:14:36.848 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:14:36.973 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:14:36.971+0000 7f571d22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:16:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:16:36.892 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:16:38.893 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:16:39.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:16:39.014+0000 7f417151e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:18:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:18:38.937 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:18:40.938 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:18:41.061 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:18:41.059+0000 7fafb859e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:20:40.980 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:20:40.981 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:20:42.982 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:20:43.107 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:20:43.104+0000 7fef4942d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:22:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:22:43.027 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:22:45.028 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:22:45.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:22:45.151+0000 7fac69b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:24:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:24:45.074 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:24:47.075 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:24:47.199 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:24:47.197+0000 7feed7d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:26:47.118 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:26:47.119 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:26:49.120 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:26:49.245 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:26:49.243+0000 7fd76866d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:28:49.164 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:28:49.165 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:28:51.165 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:28:51.290 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:28:51.288+0000 7fb142f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:30:51.210 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:30:51.211 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:30:53.212 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:30:53.347 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:30:53.345+0000 7fa5fa95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:32:53.267 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:32:53.268 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:32:55.269 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:32:55.392 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:32:55.390+0000 7f7a70c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:34:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:34:55.313 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:34:57.314 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:34:57.438 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:34:57.436+0000 7fdc86d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:36:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:36:57.358 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:36:59.358 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:36:59.482 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:36:59.480+0000 7f0ccc86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:38:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:38:59.402 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:39:01.403 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:39:01.527 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:39:01.525+0000 7f421ae2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:41:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:41:01.447 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:41:03.448 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:41:03.572 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:41:03.569+0000 7fbd71f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:43:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:43:03.492 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:43:05.493 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:43:05.617 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:43:05.614+0000 7f57be6f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:45:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:45:05.537 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:45:07.538 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:45:07.664 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:45:07.661+0000 7f529875e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:47:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:47:07.582 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:47:09.582 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:47:09.706 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:47:09.703+0000 7f0c71b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:49:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:49:09.626 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:49:11.628 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:49:11.751 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:49:11.749+0000 7fae2bd11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:51:11.670 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:51:11.671 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:51:13.673 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:51:13.796 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:51:13.793+0000 7f7b05220640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:53:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:53:13.716 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:53:15.718 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:53:15.841 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:53:15.839+0000 7fb8ecd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:55:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:55:15.763 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:55:17.764 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:55:17.889 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:55:17.886+0000 7f1d8271e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:57:17.807 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:57:17.808 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:57:19.809 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T19:57:19.933 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:57:19.930+0000 7fb943f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T19:59:19.852 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T19:59:19.853 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T19:59:21.854 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T19:59:21.978 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T19:59:21.975+0000 7f8877d04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:01:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:01:21.898 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:01:23.899 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:01:24.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:01:24.020+0000 7ff00f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:03:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:03:23.942 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:03:25.943 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:03:26.068 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:03:26.064+0000 7ff4a0b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:05:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:05:25.986 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:05:27.987 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:05:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:05:28.109+0000 7f02bc59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:07:28.030 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:07:28.031 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:07:30.032 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:07:30.157 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:07:30.154+0000 7f2da435e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:09:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:09:30.075 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:09:32.076 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:09:32.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:09:32.199+0000 7f38c1e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:11:32.118 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:11:32.119 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:11:34.121 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:11:34.245 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:11:34.241+0000 7fbd4f59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:13:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:13:34.165 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:13:36.166 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:13:36.289 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:13:36.287+0000 7fc50cf55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:15:36.208 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:15:36.209 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:15:38.210 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:15:38.334 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:15:38.330+0000 7f8fe6c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:17:38.252 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:17:38.253 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:17:40.254 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:17:40.378 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:17:40.375+0000 7f6342b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:19:40.297 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:19:40.298 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:19:42.299 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:19:42.424 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:19:42.421+0000 7f9621aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:21:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:21:42.344 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:21:44.345 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:21:44.471 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:21:44.469+0000 7f3d66504640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:23:44.390 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:23:44.390 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:23:46.391 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:23:46.515 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:23:46.511+0000 7fe8d7c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:25:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:25:46.435 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:25:48.436 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:25:48.559 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:25:48.556+0000 7fc7056de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:27:48.478 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:27:48.479 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:27:50.480 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:27:50.604 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:27:50.600+0000 7f851d4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:29:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:29:50.524 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:29:52.525 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:29:52.649 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:29:52.645+0000 7fc15ad1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:31:52.568 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:31:52.568 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:31:54.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:31:54.694 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:31:54.690+0000 7fd26686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:33:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:33:54.614 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:33:56.614 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:33:56.738 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:33:56.735+0000 7f38b1e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:35:56.657 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:35:56.657 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:35:58.659 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:35:58.782 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:35:58.779+0000 7f00d391e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:37:58.701 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:37:58.702 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:38:00.703 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:38:00.827 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:38:00.824+0000 7f6acbc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:40:00.746 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:40:00.747 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:40:02.748 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:40:02.872 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:40:02.868+0000 7f0aad82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:42:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:42:02.792 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:42:04.793 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:42:04.917 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:42:04.914+0000 7ff70066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:44:04.835 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:44:04.836 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:44:06.837 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:44:06.962 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:44:06.959+0000 7f1bc159e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:46:06.881 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:46:06.881 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:46:08.882 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:46:09.006 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:46:09.003+0000 7fc0ff51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:48:08.925 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:48:08.926 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:48:10.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:48:11.052 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:48:11.049+0000 7f2c1175e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:50:10.970 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:50:10.971 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:50:12.973 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:50:13.096 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:50:13.093+0000 7fc07860f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:52:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:52:13.016 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:52:15.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:52:15.141 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:52:15.138+0000 7f6857cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:54:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:54:15.062 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:54:17.064 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:54:17.188 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:54:17.185+0000 7f60e731e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:56:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:56:17.107 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:56:19.108 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T20:56:19.231 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:56:19.229+0000 7f6f3b31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T20:58:19.151 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T20:58:19.152 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T20:58:21.153 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T20:58:21.277 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T20:58:21.274+0000 7fd70ad9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:00:21.196 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:00:21.197 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:00:23.198 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:00:23.323 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:00:23.320+0000 7ff1bfe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:02:23.241 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:02:23.242 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:02:25.243 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:02:25.370 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:02:25.367+0000 7fa7de79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:04:25.287 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:04:25.288 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:04:27.289 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:04:27.412 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:04:27.410+0000 7fc1b375e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:06:27.332 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:06:27.333 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:06:29.334 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:06:29.458 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:06:29.455+0000 7ff933b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:08:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:08:29.378 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:08:31.388 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:08:31.514 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:08:31.511+0000 7fba04a0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:10:31.431 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:10:31.432 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:10:33.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:10:33.557 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:10:33.554+0000 7f5cbed00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:12:33.476 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:12:33.477 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:12:35.478 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:12:35.602 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:12:35.599+0000 7fe9d239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:14:35.522 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:14:35.523 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:14:37.524 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:14:37.648 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:14:37.645+0000 7f87f8f00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:16:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:16:37.568 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:16:39.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:16:39.694 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:16:39.690+0000 7f218315e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:18:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:18:39.613 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:18:41.614 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:18:41.738 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:18:41.734+0000 7faf73911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:20:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:20:41.658 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:20:43.659 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:20:43.782 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:20:43.780+0000 7f5826f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:22:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:22:43.702 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:22:45.703 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:22:45.827 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:22:45.824+0000 7f27cfb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:24:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:24:45.747 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:24:47.748 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:24:47.871 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:24:47.868+0000 7f194551e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:26:47.790 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:26:47.791 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:26:49.792 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:26:49.915 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:26:49.912+0000 7fb38699e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:28:49.834 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:28:49.848 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:28:51.849 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:28:51.973 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:28:51.970+0000 7f6f3786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:30:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:30:51.893 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:30:53.894 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:30:54.020 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:30:54.016+0000 7f3fa7a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:32:53.938 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:32:53.938 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:32:55.941 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:32:56.065 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:32:56.062+0000 7fb5ebf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:34:55.984 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:34:55.985 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:34:57.986 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:34:58.110 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:34:58.107+0000 7ff492f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:36:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:36:58.030 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:37:00.032 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:37:00.156 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:37:00.154+0000 7ff025a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:39:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:39:00.075 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:39:02.076 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:39:02.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:39:02.196+0000 7f5c3302d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:41:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:41:02.119 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:41:04.121 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:41:04.244 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:41:04.242+0000 7f65caf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:43:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:43:04.164 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:43:06.165 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:43:06.288 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:43:06.285+0000 7fd0354ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:45:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:45:06.208 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:45:08.210 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:45:08.333 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:45:08.330+0000 7fb962a20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:47:08.253 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:47:08.254 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:47:10.255 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:47:10.379 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:47:10.376+0000 7fd75006d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:49:10.298 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:49:10.299 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:49:12.300 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:49:12.424 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:49:12.422+0000 7fb73799e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:51:12.343 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:51:12.344 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:51:14.345 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:51:14.472 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:51:14.469+0000 7faec911e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:53:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:53:14.392 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:53:16.394 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:53:16.518 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:53:16.515+0000 7fa8c76ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:55:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:55:16.438 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:55:18.439 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:55:18.562 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:55:18.559+0000 7fb02ca6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:57:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:57:18.482 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:57:20.483 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T21:57:20.609 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:57:20.604+0000 7f0b0b508640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T21:59:20.527 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T21:59:20.528 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T21:59:22.530 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T21:59:22.653 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T21:59:22.651+0000 7f00ea820640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:01:22.573 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:01:22.574 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:01:24.575 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:01:24.699 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:01:24.696+0000 7f78b2cf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:03:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:03:24.619 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:03:26.619 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:03:26.743 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:03:26.740+0000 7fe73bc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:05:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:05:26.699 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:05:28.700 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:05:28.823 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:05:28.820+0000 7f412259e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:07:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:07:28.744 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:07:30.745 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:07:30.868 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:07:30.865+0000 7f38a682d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:09:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:09:30.788 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:09:32.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:09:32.913 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:09:32.910+0000 7f76f5b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:11:32.832 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:11:32.833 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:11:34.834 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:11:34.960 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:11:34.954+0000 7ff3c1d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:13:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:13:34.878 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:13:36.879 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:13:37.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:13:37.001+0000 7f892806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:15:36.921 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:15:36.922 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:15:38.923 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:15:39.048 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:15:39.044+0000 7ffa85a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:17:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:17:38.967 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:17:40.969 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:17:41.093 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:17:41.090+0000 7f6b1fb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:19:41.012 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:19:41.013 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:19:43.014 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:19:43.138 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:19:43.135+0000 7f150695e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:21:43.057 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:21:43.058 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:21:45.060 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:21:45.184 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:21:45.182+0000 7feae0f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:23:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:23:45.104 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:23:47.106 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:23:47.229 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:23:47.225+0000 7fe37179e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:25:47.148 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:25:47.149 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:25:49.150 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:25:49.274 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:25:49.271+0000 7f1b8131e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:27:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:27:49.194 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:27:51.195 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:27:51.320 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:27:51.317+0000 7f2528e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:29:51.238 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:29:51.239 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:29:53.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:29:53.364 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:29:53.361+0000 7f2defb11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:31:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:31:54.020 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:31:56.022 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:31:56.146 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:31:56.143+0000 7fb680c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:33:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:33:58.055 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:34:00.057 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:34:00.180 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:34:00.178+0000 7f3530d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:36:00.100 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:36:04.647 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:36:06.648 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:36:06.772 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:36:06.769+0000 7f5fc26f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:38:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:38:06.692 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:38:08.693 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:38:08.817 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:38:08.814+0000 7fe20ac60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:40:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:40:08.736 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:40:10.738 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:40:10.863 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:40:10.859+0000 7fbe616de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:42:10.780 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:42:10.781 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:42:12.783 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:42:12.906 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:42:12.904+0000 7f80bdb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:44:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:44:12.826 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:44:14.827 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:44:14.951 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:44:14.947+0000 7f4fc166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:46:14.869 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:46:14.870 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:46:16.871 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:46:16.994 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:46:16.992+0000 7fc07b31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:48:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:48:16.915 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:48:18.916 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:48:19.040 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:48:19.036+0000 7f5bdd66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:50:18.959 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:50:18.960 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:50:20.961 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:50:21.085 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:50:21.083+0000 7f609646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:52:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:52:21.005 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:52:23.006 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:52:23.130 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:52:23.126+0000 7f8d0946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:54:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:54:23.049 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:54:25.050 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:54:25.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:54:25.172+0000 7f9d31cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:56:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:56:25.094 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:56:27.094 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T22:56:27.218 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:56:27.215+0000 7f0c7302d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T22:58:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T22:58:27.139 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T22:58:29.140 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T22:58:29.263 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T22:58:29.260+0000 7f5400b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:00:29.183 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:00:29.183 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:00:31.184 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:00:31.308 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:00:31.306+0000 7fb4aa417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:02:31.227 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:02:31.243 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:02:33.244 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:02:33.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:02:33.365+0000 7f6703c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:04:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:04:33.287 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:04:35.288 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:04:35.412 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:04:35.409+0000 7f5e7b555640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:06:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:06:35.332 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:06:37.333 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:06:37.457 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:06:37.454+0000 7faa830ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:08:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:08:37.376 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:08:39.377 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:08:39.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:08:39.497+0000 7fa35326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:10:39.419 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:10:39.421 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:10:41.422 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:10:41.546 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:10:41.543+0000 7fb32f59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:12:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:12:41.466 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:12:43.467 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:12:43.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:12:43.588+0000 7f8aef59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:14:43.509 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:14:43.510 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:14:45.511 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:14:45.635 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:14:45.633+0000 7f09b239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:16:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:16:45.555 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:16:47.557 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:16:47.682 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:16:47.680+0000 7f6fc8a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:18:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:18:47.601 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:18:49.602 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:18:49.726 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:18:49.723+0000 7f504d59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:20:49.644 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:20:49.646 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:20:51.647 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:20:51.772 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:20:51.770+0000 7fa7cb624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:22:51.690 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:22:51.691 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:22:53.692 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:22:53.816 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:22:53.813+0000 7f8b0055e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:24:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:24:53.736 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:24:55.736 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:24:55.860 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:24:55.857+0000 7f5fcb55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:26:55.779 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:26:55.780 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:26:57.781 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:26:57.904 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:26:57.902+0000 7f387226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:28:57.823 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:28:57.824 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:28:59.826 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:28:59.949 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:28:59.947+0000 7f52f826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:30:59.868 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:30:59.869 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:31:01.870 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:31:01.994 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:31:01.992+0000 7fc5b339e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:33:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:33:01.914 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:33:03.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:33:04.039 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:33:04.037+0000 7f502b504640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:35:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:35:03.959 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:35:05.961 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:35:06.084 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:35:06.082+0000 7f09a106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:37:06.004 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:37:06.004 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:37:08.005 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:37:08.129 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:37:08.127+0000 7f709839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:39:08.048 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:39:08.049 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:39:10.051 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:39:10.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:39:10.171+0000 7f7585e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:41:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:41:10.094 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:41:12.096 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:41:12.219 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:41:12.217+0000 7f36a386d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:43:12.139 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:43:12.139 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:43:14.141 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:43:14.266 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:43:14.263+0000 7f84ab71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:45:14.184 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:45:14.185 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:45:16.186 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:45:16.311 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:45:16.308+0000 7fd48499e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:47:16.228 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:47:16.229 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:47:18.230 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:47:18.354 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:47:18.350+0000 7f59492de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:49:18.272 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:49:18.273 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:49:20.274 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:49:20.398 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:49:20.395+0000 7f877995e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:51:20.316 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:51:20.317 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:51:22.318 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:51:22.441 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:51:22.438+0000 7ff8a1017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:53:22.361 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:53:22.362 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:53:24.363 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:53:24.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:53:24.484+0000 7f2c63b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:55:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:55:24.406 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:55:26.407 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:55:26.531 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:55:26.527+0000 7f067166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:57:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:57:26.450 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:57:28.451 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-28T23:57:28.575 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:57:28.573+0000 7ff23822d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T23:59:28.495 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T23:59:28.496 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-28T23:59:30.497 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-28T23:59:30.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-28T23:59:30.620+0000 7f9db2500640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:01:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:01:30.543 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:01:32.544 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:01:32.671 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:01:32.668+0000 7f2f6506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:03:32.590 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:03:32.591 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:03:34.593 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:03:34.716 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:03:34.714+0000 7ff5772de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:05:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:05:34.637 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:05:36.638 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:05:36.762 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:05:36.759+0000 7fb03595e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:07:36.681 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:07:36.682 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:07:38.684 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:07:38.807 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:07:38.805+0000 7f1f0920f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:09:38.727 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:09:38.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:09:40.730 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:09:40.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:09:40.850+0000 7f77e326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:11:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:11:40.773 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:11:42.775 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:11:42.899 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:11:42.896+0000 7fed2702d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:13:42.818 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:13:42.819 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:13:44.820 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:13:44.944 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:13:44.942+0000 7f8f1dc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:15:44.863 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:15:44.864 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:15:46.865 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:15:46.988 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:15:46.986+0000 7f4e5495e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:17:46.907 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:17:46.908 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:17:48.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:17:49.033 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:17:49.030+0000 7f9829908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:19:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:19:48.953 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:19:50.954 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:19:51.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:19:51.075+0000 7fccee71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:21:50.997 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:21:50.998 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:21:52.999 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:21:53.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:21:53.120+0000 7f4c32c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:23:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:23:54.273 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:23:56.274 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:23:56.397 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:23:56.395+0000 7fb2b9e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:25:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:25:56.318 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:25:58.319 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:25:58.444 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:25:58.442+0000 7faed559e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:27:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:27:58.364 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:28:00.365 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:28:00.488 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:28:00.486+0000 7f74e3c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:30:00.407 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:30:00.408 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:30:02.410 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:30:02.535 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:30:02.532+0000 7fb5c7b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:32:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:32:02.453 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:32:04.455 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:32:04.578 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:32:04.576+0000 7fa30899e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:34:04.497 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:34:04.498 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:34:06.499 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:34:06.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:34:06.621+0000 7fdebf99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:36:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:36:06.544 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:36:08.545 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:36:08.670 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:36:08.667+0000 7efccdf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:38:08.587 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:38:08.588 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:38:10.589 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:38:10.713 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:38:10.711+0000 7f382566d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:40:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:40:10.633 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:40:12.634 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:40:12.758 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:40:12.755+0000 7febd2b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:42:12.677 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:42:12.678 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:42:14.679 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:42:14.803 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:42:14.800+0000 7efdb660f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:44:14.722 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:44:14.723 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:44:16.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:44:16.848 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:44:16.846+0000 7f7459806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:46:16.767 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:46:16.767 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:46:18.769 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:46:18.893 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:46:18.890+0000 7f99f891e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:48:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:48:18.812 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:48:20.813 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:48:20.937 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:48:20.935+0000 7fdc05d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:50:20.856 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:50:20.857 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:50:22.858 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:50:22.981 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:50:22.979+0000 7fdf2e46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:52:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:52:22.901 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:52:24.903 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:52:25.028 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:52:25.026+0000 7fcb7c02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:54:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:54:24.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:54:26.947 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:54:27.069 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:54:27.067+0000 7fcf77c17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:56:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:56:26.990 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:56:28.992 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T00:56:29.116 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:56:29.114+0000 7fdb1046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T00:58:29.035 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T00:58:29.036 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T00:58:31.037 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T00:58:31.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T00:58:31.158+0000 7f3b4d39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:00:31.080 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:00:31.081 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:00:33.082 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:00:33.205 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:00:33.203+0000 7f71b179e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:02:33.124 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:02:33.125 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:02:35.128 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:02:35.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:02:35.250+0000 7fee7aa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:04:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:04:35.173 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:04:37.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:04:37.297 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:04:37.296+0000 7f71bbade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:06:37.217 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:06:37.218 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:06:39.220 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:06:39.345 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:06:39.344+0000 7f63d2217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:08:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:08:39.263 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:08:41.265 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:08:41.388 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:08:41.386+0000 7f895ef04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:10:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:10:41.309 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:10:43.310 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:10:43.433 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:10:43.432+0000 7fc8b235e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:12:43.353 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:12:43.354 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:12:45.355 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:12:45.480 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:12:45.478+0000 7fe2e6c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:14:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:14:45.400 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:14:47.402 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:14:47.525 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:14:47.523+0000 7f09f7a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:16:47.444 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:16:47.445 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:16:49.446 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:16:49.569 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:16:49.567+0000 7f4b6215e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:18:49.488 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:18:49.489 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:18:51.490 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:18:51.614 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:18:51.613+0000 7f33b346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:20:51.533 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:20:51.534 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:20:53.535 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:20:53.661 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:20:53.658+0000 7fc4c7704640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:22:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:22:53.583 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:22:55.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:22:55.709 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:22:55.707+0000 7f8a21f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:24:55.629 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:24:55.630 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:24:57.631 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:24:57.755 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:24:57.753+0000 7f56a226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:26:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:26:57.674 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:26:59.675 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:26:59.799 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:26:59.797+0000 7ff8f0660640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:28:59.718 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:28:59.719 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:29:01.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:29:01.845 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:29:01.842+0000 7f0105c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:31:01.763 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:31:01.764 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:31:03.765 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:31:03.890 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:31:03.887+0000 7fc997955640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:33:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:33:03.809 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:33:05.810 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:33:05.933 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:33:05.929+0000 7f9403c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:35:05.852 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:35:05.853 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:35:07.854 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:35:07.978 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:35:07.975+0000 7fbadfa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:37:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:37:07.898 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:37:09.900 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:37:10.024 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:37:10.021+0000 7f9aa286d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:39:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:39:09.944 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:39:11.945 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:39:12.068 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:39:12.064+0000 7fc7b3a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:41:11.989 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:41:11.990 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:41:13.991 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:41:14.115 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:41:14.112+0000 7f5d5c79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:43:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:43:14.036 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:43:16.041 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:43:16.165 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:43:16.161+0000 7f8037a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:45:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:45:16.086 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:45:18.088 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:45:18.211 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:45:18.208+0000 7f39b419e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:47:18.130 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:47:18.131 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:47:20.131 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:47:20.255 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:47:20.251+0000 7fdf2dc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:49:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:49:20.176 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:49:22.177 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:49:22.302 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:49:22.299+0000 7f11b522d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:51:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:51:22.220 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:51:24.221 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:51:24.344 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:51:24.340+0000 7f563715e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:53:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:53:24.264 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:53:26.265 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:53:26.388 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:53:26.385+0000 7f1929b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:55:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:55:26.308 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:55:28.309 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:55:28.435 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:55:28.432+0000 7f49ff46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:57:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:57:28.353 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:57:30.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T01:57:30.477 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:57:30.474+0000 7fbddce60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T01:59:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T01:59:30.397 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T01:59:32.398 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T01:59:32.521 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T01:59:32.518+0000 7f12a851e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T02:01:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T02:01:32.441 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T02:01:34.443 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T02:01:34.566 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T02:01:34.562+0000 7fb4ca6f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T02:03:34.485 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T02:03:34.486 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T02:03:36.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T02:03:36.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T02:03:36.608+0000 7f74a3a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T02:05:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T02:05:36.531 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T02:05:38.532 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T02:05:38.656 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T02:05:38.652+0000 7f07ab55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T02:07:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T02:07:38.577 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T02:07:40.578 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T02:07:40.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T02:07:40.699+0000 7fb0e0cde640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T02:09:40.621 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T02:09:40.621 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T02:09:42.623 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T02:09:42.746 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T02:09:42.743+0000 7f88436de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T02:11:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T02:11:42.670 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T02:11:44.672 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T02:11:44.798 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T02:11:44.795+0000 7f68ede6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T02:13:44.717 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T02:13:44.717 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T02:13:46.719 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T02:13:46.843 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T02:13:46.840+0000 7f079bf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T02:15:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T02:15:46.764 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T02:15:48.766 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-29T02:15:48.890 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T02:15:48.886+0000 7f973c75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T02:17:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-29T02:17:48.810 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-29T02:17:50.810 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-29T02:17:50.934 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-29T02:17:50.931+0000 7ff2a0d15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T02:18:46.545 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-11-29T02:18:46.545 DEBUG:teuthology.task.console_log:Killing console logger for smithi055 2023-11-29T02:18:46.546 DEBUG:teuthology.task.console_log:Killing console logger for smithi060 2023-11-29T02:18:46.546 DEBUG:teuthology.exit:Finished running handlers