2024-04-25T09:16:13.908 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T09:16:13.908 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi/7672977 --verbose --owner scheduled_matan@teuthology --description crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} --name matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi -- /home/teuthworker/archive/matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi/7672977/orig.config.yaml 2024-04-25T09:16:13.939 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T09:16:14.026 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi/7672977 branch: wip-matanb-crimson-testing-april-22 description: crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} email: null first_in_suite: false job_id: '7672977' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-testing-april-22 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(PG_ - \(OSD_ - \(OBJECT_ - \(POOL_APP_NOT_ENABLED\) sha1: a126334d8733673d119ffdf40d6ae4b541a0a1cd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: extra_system_packages: - python3-pytest flavor: crimson sha1: a126334d8733673d119ffdf40d6ae4b541a0a1cd selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 472cc2b524a88306ef4535053d26658c5fd6e9db owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 1211 sha1: a126334d8733673d119ffdf40d6ae4b541a0a1cd sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_472cc2b524a88306ef4535053d26658c5fd6e9db/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 472cc2b524a88306ef4535053d26658c5fd6e9db targets: smithi043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ/waeR5lfqj8M3qpqR8FrH+LBOPZPD4bWUQFEmy26+gU96AWDxesKNhELsdhnzJBJyJTGZG+p+ff4mUgcF9jVY= smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMITCQg/P/TXatLUkWpLdWOo+PWhMEbA40XRXgSSf2pNhLX2C2SRiY5OGznGrRQ5ieeVutHA2JKHKLiIi31Llzg= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rados/test_python.sh -m 'not (wait or tier or ec or bench or stats)' timeout: 1h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_08:07:11 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T09:16:14.026 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_472cc2b524a88306ef4535053d26658c5fd6e9db/qa; will attempt to use it 2024-04-25T09:16:14.027 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_472cc2b524a88306ef4535053d26658c5fd6e9db/qa/tasks 2024-04-25T09:16:14.027 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T09:16:14.028 INFO:teuthology.task.internal:Checking packages... 2024-04-25T09:16:14.053 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'a126334d8733673d119ffdf40d6ae4b541a0a1cd' 2024-04-25T09:16:14.054 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T09:16:14.054 INFO:teuthology.packaging:ref: None 2024-04-25T09:16:14.054 INFO:teuthology.packaging:tag: None 2024-04-25T09:16:14.054 INFO:teuthology.packaging:branch: wip-matanb-crimson-testing-april-22 2024-04-25T09:16:14.054 INFO:teuthology.packaging:sha1: a126334d8733673d119ffdf40d6ae4b541a0a1cd 2024-04-25T09:16:14.054 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-testing-april-22 2024-04-25T09:16:14.195 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3194.ga126334d 2024-04-25T09:16:14.196 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T09:16:14.202 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T09:16:14.203 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T09:16:14.276 INFO:teuthology.task.internal:Saving configuration 2024-04-25T09:16:14.288 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T09:16:14.294 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T09:16:14.320 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi/7672977', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 09:10:58.600630', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ/waeR5lfqj8M3qpqR8FrH+LBOPZPD4bWUQFEmy26+gU96AWDxesKNhELsdhnzJBJyJTGZG+p+ff4mUgcF9jVY='} 2024-04-25T09:16:14.345 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi/7672977', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 09:10:58.599562', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMITCQg/P/TXatLUkWpLdWOo+PWhMEbA40XRXgSSf2pNhLX2C2SRiY5OGznGrRQ5ieeVutHA2JKHKLiIi31Llzg='} 2024-04-25T09:16:14.346 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T09:16:14.352 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-04-25T09:16:14.352 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-25T09:16:14.352 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T09:16:14.422 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fada6616ee0>, signals=[15]) 2024-04-25T09:16:14.422 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T09:16:14.429 INFO:teuthology.task.internal:Opening connections... 2024-04-25T09:16:14.429 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2024-04-25T09:16:14.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T09:16:14.511 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2024-04-25T09:16:14.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T09:16:14.589 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T09:16:14.595 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2024-04-25T09:16:14.614 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2024-04-25T09:16:14.614 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2024-04-25T09:16:14.672 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="CentOS Stream" 2024-04-25T09:16:14.673 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="9" 2024-04-25T09:16:14.673 INFO:teuthology.orchestra.run.smithi043.stdout:ID="centos" 2024-04-25T09:16:14.673 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE="rhel fedora" 2024-04-25T09:16:14.673 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="9" 2024-04-25T09:16:14.673 INFO:teuthology.orchestra.run.smithi043.stdout:PLATFORM_ID="platform:el9" 2024-04-25T09:16:14.673 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T09:16:14.673 INFO:teuthology.orchestra.run.smithi043.stdout:ANSI_COLOR="0;31" 2024-04-25T09:16:14.673 INFO:teuthology.orchestra.run.smithi043.stdout:LOGO="fedora-logo-icon" 2024-04-25T09:16:14.673 INFO:teuthology.orchestra.run.smithi043.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T09:16:14.673 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://centos.org/" 2024-04-25T09:16:14.673 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T09:16:14.673 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T09:16:14.674 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T09:16:14.674 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2024-04-25T09:16:14.702 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2024-04-25T09:16:14.720 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2024-04-25T09:16:14.720 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2024-04-25T09:16:14.776 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2024-04-25T09:16:14.776 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="9" 2024-04-25T09:16:14.776 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2024-04-25T09:16:14.776 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2024-04-25T09:16:14.776 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="9" 2024-04-25T09:16:14.777 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el9" 2024-04-25T09:16:14.777 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T09:16:14.777 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2024-04-25T09:16:14.777 INFO:teuthology.orchestra.run.smithi164.stdout:LOGO="fedora-logo-icon" 2024-04-25T09:16:14.777 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T09:16:14.777 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2024-04-25T09:16:14.777 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T09:16:14.777 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T09:16:14.777 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T09:16:14.777 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2024-04-25T09:16:14.802 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T09:16:14.811 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T09:16:14.821 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T09:16:14.821 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2024-04-25T09:16:14.824 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2024-04-25T09:16:14.839 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T09:16:14.844 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T09:16:14.844 DEBUG:teuthology.orchestra.run.smithi043:> test -z $(ls -A /var/lib/ceph) 2024-04-25T09:16:14.882 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2024-04-25T09:16:14.996 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T09:16:15.051 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T09:16:15.065 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-25T09:16:15.065 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-25T09:16:15.065 DEBUG:teuthology.orchestra.run.smithi043:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T09:16:15.066 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T09:16:15.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T09:16:15.082 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-04-25T09:16:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T09:16:15.083 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-04-25T09:16:15.139 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-437.el9.x86_64 2024-04-25T09:16:15.140 INFO:teuthology.task.kernel:Running kernel on smithi164: 5.14.0-437.el9.x86_64 2024-04-25T09:16:15.140 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum install -y kernel 2024-04-25T09:16:15.142 INFO:teuthology.orchestra.run.smithi043.stdout:5.14.0-437.el9.x86_64 2024-04-25T09:16:15.142 INFO:teuthology.task.kernel:Running kernel on smithi043: 5.14.0-437.el9.x86_64 2024-04-25T09:16:15.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum install -y kernel 2024-04-25T09:16:17.374 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-25T09:16:17.375 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-25T09:16:19.310 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-25T09:16:20.525 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 19 MB 00:02 2024-04-25T09:16:24.068 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.2 MB 00:00 2024-04-25T09:16:25.234 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 6.0 MB/s | 6.2 MB 00:01 2024-04-25T09:16:25.927 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-25T09:16:26.709 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 16 kB 00:00 2024-04-25T09:16:27.545 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-25T09:16:32.503 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-25T09:16:34.385 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T09:16:34.385 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T09:16:34.472 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-04-25T09:16:34.473 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-04-25T09:16:34.473 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-04-25T09:16:34.584 DEBUG:teuthology.orchestra.run.smithi164:> echo no | sudo yum reinstall kernel || true 2024-04-25T09:16:35.124 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 09:16:32 AM UTC. 2024-04-25T09:16:35.251 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-04-25T09:16:35.262 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-04-25T09:16:35.262 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-04-25T09:16:35.262 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-04-25T09:16:35.262 INFO:teuthology.orchestra.run.smithi164.stdout:Reinstalling: 2024-04-25T09:16:35.262 INFO:teuthology.orchestra.run.smithi164.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T09:16:35.262 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:16:35.262 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-04-25T09:16:35.262 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-04-25T09:16:35.262 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:16:35.262 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 255 k 2024-04-25T09:16:35.262 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 0 2024-04-25T09:16:35.263 INFO:teuthology.orchestra.run.smithi164.stderr:Operation aborted. 2024-04-25T09:16:35.320 INFO:teuthology.orchestra.run.smithi164.stdout:Is this ok [y/N]: 2024-04-25T09:16:35.320 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum reinstall -y kernel || true 2024-04-25T09:16:35.815 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 09:16:32 AM UTC. 2024-04-25T09:16:35.931 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-04-25T09:16:35.932 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-04-25T09:16:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-04-25T09:16:35.932 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-04-25T09:16:35.932 INFO:teuthology.orchestra.run.smithi164.stdout:Reinstalling: 2024-04-25T09:16:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T09:16:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:16:35.932 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-04-25T09:16:35.933 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-04-25T09:16:35.933 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:16:35.933 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 255 k 2024-04-25T09:16:35.933 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 0 2024-04-25T09:16:35.933 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-04-25T09:16:36.247 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-25T09:16:36.247 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-04-25T09:16:36.248 INFO:teuthology.orchestra.run.smithi164.stdout:Total 809 kB/s | 255 kB 00:00 2024-04-25T09:16:36.263 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-04-25T09:16:36.444 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-04-25T09:16:36.444 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-04-25T09:16:36.449 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-04-25T09:16:36.449 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-04-25T09:16:36.574 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-04-25T09:16:36.656 INFO:teuthology.orchestra.run.smithi164.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T09:16:36.657 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T09:16:37.365 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T09:16:37.365 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T09:16:38.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T09:16:38.053 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:16:38.053 INFO:teuthology.orchestra.run.smithi164.stdout:Reinstalled: 2024-04-25T09:16:38.053 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T09:16:38.053 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:16:38.053 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-04-25T09:16:38.246 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T09:16:38.294 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T09:16:38.294 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T09:16:38.294 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T09:16:38.294 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-25T09:16:38.294 DEBUG:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T09:16:38.322 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2024-04-25T09:16:38.347 DEBUG:teuthology.parallel:result is None 2024-04-25T09:16:46.937 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-04-25T09:16:52.200 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-25T09:16:54.173 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T09:16:54.173 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T09:16:54.271 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T09:16:54.272 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-25T09:16:54.272 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T09:16:54.375 DEBUG:teuthology.orchestra.run.smithi043:> echo no | sudo yum reinstall kernel || true 2024-04-25T09:16:54.897 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:02 ago on Thu 25 Apr 2024 09:16:52 AM UTC. 2024-04-25T09:16:55.020 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T09:16:55.029 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T09:16:55.030 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-04-25T09:16:55.030 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T09:16:55.030 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalling: 2024-04-25T09:16:55.030 INFO:teuthology.orchestra.run.smithi043.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T09:16:55.030 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:16:55.030 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T09:16:55.030 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T09:16:55.030 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:16:55.030 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 255 k 2024-04-25T09:16:55.030 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 0 2024-04-25T09:16:55.031 INFO:teuthology.orchestra.run.smithi043.stderr:Operation aborted. 2024-04-25T09:16:55.094 INFO:teuthology.orchestra.run.smithi043.stdout:Is this ok [y/N]: 2024-04-25T09:16:55.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum reinstall -y kernel || true 2024-04-25T09:16:55.610 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 09:16:52 AM UTC. 2024-04-25T09:16:55.730 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T09:16:55.731 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T09:16:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-04-25T09:16:55.731 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T09:16:55.731 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalling: 2024-04-25T09:16:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T09:16:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:16:55.732 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T09:16:55.732 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T09:16:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:16:55.732 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 255 k 2024-04-25T09:16:55.732 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 0 2024-04-25T09:16:55.732 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T09:16:56.119 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-5.14.0-437.el9.x86_64.rpm 862 kB/s | 255 kB 00:00 2024-04-25T09:16:56.119 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T09:16:56.119 INFO:teuthology.orchestra.run.smithi043.stdout:Total 659 kB/s | 255 kB 00:00 2024-04-25T09:16:56.138 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T09:16:56.347 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T09:16:56.347 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T09:16:56.352 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T09:16:56.352 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T09:16:56.481 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T09:16:56.561 INFO:teuthology.orchestra.run.smithi043.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T09:16:56.562 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T09:16:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T09:16:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T09:16:57.911 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T09:16:57.911 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:16:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalled: 2024-04-25T09:16:57.912 INFO:teuthology.orchestra.run.smithi043.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T09:16:57.912 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:16:57.912 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T09:16:58.118 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T09:16:58.167 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T09:16:58.167 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T09:16:58.167 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T09:16:58.167 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-25T09:16:58.168 DEBUG:teuthology.orchestra.run.smithi043:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T09:16:58.198 INFO:teuthology.orchestra.run.smithi043.stdout:ttyS1 2024-04-25T09:16:58.225 DEBUG:teuthology.parallel:result is None 2024-04-25T09:16:58.226 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T09:16:58.234 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T09:16:58.234 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T09:16:58.268 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T09:16:58.287 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T09:16:58.296 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T09:16:58.310 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T09:16:58.310 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T09:16:58.323 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T09:16:58.427 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T09:16:58.437 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T09:16:58.438 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T09:16:58.440 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T09:16:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:16:58.478 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:16:58.481 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:16:58.489 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:16:58.490 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T09:16:58.502 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T09:16:58.502 DEBUG:teuthology.orchestra.run.smithi043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T09:16:58.526 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T09:16:58.557 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T09:16:58.571 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T09:16:58.571 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T09:16:58.590 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T09:16:58.613 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T09:16:58.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T09:16:58.749 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T09:16:58.790 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T09:16:58.836 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:16:58.836 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T09:16:58.894 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T09:16:58.939 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T09:16:58.988 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T09:16:59.071 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T09:16:59.111 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:16:59.112 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T09:16:59.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo service rsyslog restart 2024-04-25T09:16:59.179 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2024-04-25T09:16:59.214 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T09:16:59.256 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T09:16:59.565 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T09:16:59.574 INFO:teuthology.task.internal:Starting timer... 2024-04-25T09:16:59.574 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T09:16:59.592 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T09:16:59.608 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-25T09:16:59.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo service auditd rotate 2024-04-25T09:16:59.675 INFO:teuthology.orchestra.run.smithi043.stdout:Rotating logs: 2024-04-25T09:16:59.677 DEBUG:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2024-04-25T09:16:59.742 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: 2024-04-25T09:16:59.744 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T09:16:59.744 DEBUG:teuthology.orchestra.run.smithi043:> /usr/sbin/getenforce 2024-04-25T09:16:59.768 INFO:teuthology.orchestra.run.smithi043.stdout:Permissive 2024-04-25T09:16:59.768 DEBUG:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2024-04-25T09:16:59.790 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2024-04-25T09:16:59.790 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi043.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi164.front.sepia.ceph.com': 'permissive'} 2024-04-25T09:16:59.791 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T09:16:59.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T09:16:59.834 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T09:16:59.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T09:16:59.859 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T09:16:59.859 DEBUG:teuthology.orchestra.run.smithi043:> sudo /usr/sbin/setenforce permissive 2024-04-25T09:16:59.900 DEBUG:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2024-04-25T09:16:59.924 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T09:16:59.935 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T09:16:59.942 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T09:17:00.073 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T09:17:00.090 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T09:17:00.091 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi043.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T09:20:31.571 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi043.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2024-04-25T09:20:31.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-04-25T09:20:31.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T09:20:31.649 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-04-25T09:20:31.729 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-04-25T09:20:31.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-04-25T09:20:31.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T09:20:31.805 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-04-25T09:20:31.890 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-04-25T09:20:31.890 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T09:20:31.902 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T09:20:31.902 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T09:20:31.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T09:20:31.905 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T09:20:31.905 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T09:20:31.931 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T09:20:31.945 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T09:20:31.959 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T09:20:31.968 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: ntpd: command not found 2024-04-25T09:20:31.972 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T09:20:31.979 INFO:teuthology.orchestra.run.smithi043.stdout:506 Cannot talk to daemon 2024-04-25T09:20:31.993 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T09:20:31.997 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2024-04-25T09:20:32.006 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T09:20:32.008 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2024-04-25T09:20:32.021 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T09:20:32.035 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T09:20:32.061 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-04-25T09:20:32.065 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T09:20:32.065 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-04-25T09:20:32.065 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T09:20:32.065 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T09:20:32.065 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T09:20:32.065 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T09:20:32.086 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-04-25T09:20:32.090 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T09:20:32.090 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-04-25T09:20:32.090 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T09:20:32.090 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T09:20:32.090 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T09:20:32.090 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T09:20:32.090 INFO:teuthology.run_tasks:Running task install... 2024-04-25T09:20:32.098 DEBUG:teuthology.task.install:project ceph 2024-04-25T09:20:32.099 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_system_packages': ['python3-pytest'], 'flavor': 'crimson', 'sha1': 'a126334d8733673d119ffdf40d6ae4b541a0a1cd'}} 2024-04-25T09:20:32.099 DEBUG:teuthology.task.install:config {'extra_system_packages': ['python3-pytest'], 'flavor': 'crimson', 'sha1': 'a126334d8733673d119ffdf40d6ae4b541a0a1cd'} 2024-04-25T09:20:32.099 INFO:teuthology.task.install:Using flavor: crimson 2024-04-25T09:20:32.104 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T09:20:32.104 INFO:teuthology.task.install:extra packages: [] 2024-04-25T09:20:32.104 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': ['python3-pytest'], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a126334d8733673d119ffdf40d6ae4b541a0a1cd', 'tag': None, 'wait_for_package': False} 2024-04-25T09:20:32.105 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=a126334d8733673d119ffdf40d6ae4b541a0a1cd 2024-04-25T09:20:32.107 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': ['python3-pytest'], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a126334d8733673d119ffdf40d6ae4b541a0a1cd', 'tag': None, 'wait_for_package': False} 2024-04-25T09:20:32.107 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=a126334d8733673d119ffdf40d6ae4b541a0a1cd 2024-04-25T09:20:32.245 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-april-22/a126334d8733673d119ffdf40d6ae4b541a0a1cd/centos/9/flavors/crimson/ 2024-04-25T09:20:32.245 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3194.ga126334d 2024-04-25T09:20:32.260 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-april-22/a126334d8733673d119ffdf40d6ae4b541a0a1cd/centos/9/flavors/crimson/ 2024-04-25T09:20:32.260 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3194.ga126334d 2024-04-25T09:20:32.355 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-april-22/a126334d8733673d119ffdf40d6ae4b541a0a1cd/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-april-22/a126334d8733673d119ffdf40d6ae4b541a0a1cd/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-april-22/a126334d8733673d119ffdf40d6ae4b541a0a1cd/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T09:20:32.355 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:20:32.355 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T09:20:32.373 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-april-22/a126334d8733673d119ffdf40d6ae4b541a0a1cd/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-april-22/a126334d8733673d119ffdf40d6ae4b541a0a1cd/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-april-22/a126334d8733673d119ffdf40d6ae4b541a0a1cd/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T09:20:32.373 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:20:32.373 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T09:20:32.380 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-pytest, python3-pytest, python3-pytest on remote rpm x86_64 2024-04-25T09:20:32.381 DEBUG:teuthology.orchestra.run.smithi043:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a126334d8733673d119ffdf40d6ae4b541a0a1cd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T09:20:32.397 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-pytest, python3-pytest, python3-pytest, python3-pytest on remote rpm x86_64 2024-04-25T09:20:32.397 DEBUG:teuthology.orchestra.run.smithi164:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a126334d8733673d119ffdf40d6ae4b541a0a1cd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T09:20:32.446 DEBUG:teuthology.orchestra.run.smithi043:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T09:20:32.461 DEBUG:teuthology.orchestra.run.smithi164:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T09:20:32.521 DEBUG:teuthology.orchestra.run.smithi043:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T09:20:32.535 DEBUG:teuthology.orchestra.run.smithi164:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T09:20:32.584 INFO:teuthology.orchestra.run.smithi043.stdout:check_obsoletes = 1 2024-04-25T09:20:32.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum clean all 2024-04-25T09:20:32.598 INFO:teuthology.orchestra.run.smithi164.stdout:check_obsoletes = 1 2024-04-25T09:20:32.600 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum clean all 2024-04-25T09:20:32.821 INFO:teuthology.orchestra.run.smithi043.stdout:48 files removed 2024-04-25T09:20:32.831 INFO:teuthology.orchestra.run.smithi164.stdout:48 files removed 2024-04-25T09:20:32.847 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-pytest python3-pytest python3-pytest python3-pytest 2024-04-25T09:20:32.860 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-pytest python3-pytest python3-pytest python3-pytest 2024-04-25T09:20:33.461 INFO:teuthology.orchestra.run.smithi043.stdout:ceph packages for x86_64 255 kB/s | 87 kB 00:00 2024-04-25T09:20:33.702 INFO:teuthology.orchestra.run.smithi164.stdout:ceph packages for x86_64 153 kB/s | 87 kB 00:00 2024-04-25T09:20:33.795 INFO:teuthology.orchestra.run.smithi043.stdout:ceph noarch packages 41 kB/s | 12 kB 00:00 2024-04-25T09:20:34.154 INFO:teuthology.orchestra.run.smithi043.stdout:ceph source packages 6.3 kB/s | 2.0 kB 00:00 2024-04-25T09:20:34.278 INFO:teuthology.orchestra.run.smithi164.stdout:ceph noarch packages 22 kB/s | 12 kB 00:00 2024-04-25T09:20:34.670 INFO:teuthology.orchestra.run.smithi164.stdout:ceph source packages 5.7 kB/s | 2.0 kB 00:00 2024-04-25T09:20:35.228 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-25T09:20:35.938 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-04-25T09:20:37.930 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-04-25T09:20:38.254 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 19 MB 00:01 2024-04-25T09:20:42.773 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 6.2 MB 00:01 2024-04-25T09:20:43.230 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.2 MB 00:01 2024-04-25T09:20:44.498 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-25T09:20:44.806 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-25T09:20:45.995 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-25T09:20:46.226 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-25T09:20:51.274 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-25T09:20:51.342 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-25T09:20:53.404 INFO:teuthology.orchestra.run.smithi164.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T09:20:53.404 INFO:teuthology.orchestra.run.smithi164.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T09:20:53.465 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-04-25T09:20:53.473 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================= 2024-04-25T09:20:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-04-25T09:20:53.473 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================= 2024-04-25T09:20:53.473 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-04-25T09:20:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: ceph x86_64 2:19.0.0-3194.ga126334d.el9 ceph 6.5 k 2024-04-25T09:20:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base x86_64 2:19.0.0-3194.ga126334d.el9 ceph 5.9 M 2024-04-25T09:20:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse x86_64 2:19.0.0-3194.ga126334d.el9 ceph 870 k 2024-04-25T09:20:53.474 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3194.ga126334d.el9 ceph 157 k 2024-04-25T09:20:53.474 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr x86_64 2:19.0.0-3194.ga126334d.el9 ceph 1.6 M 2024-04-25T09:20:53.474 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 142 k 2024-04-25T09:20:53.474 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 3.5 M 2024-04-25T09:20:53.474 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 7.4 M 2024-04-25T09:20:53.474 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 49 k 2024-04-25T09:20:53.474 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw x86_64 2:19.0.0-3194.ga126334d.el9 ceph 10 M 2024-04-25T09:20:53.474 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test x86_64 2:19.0.0-3194.ga126334d.el9 ceph 84 M 2024-04-25T09:20:53.474 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 273 k 2024-04-25T09:20:53.474 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 762 k 2024-04-25T09:20:53.474 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel x86_64 2:19.0.0-3194.ga126334d.el9 ceph 32 k 2024-04-25T09:20:53.475 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2 x86_64 2:19.0.0-3194.ga126334d.el9 ceph 796 k 2024-04-25T09:20:53.475 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel x86_64 2:19.0.0-3194.ga126334d.el9 ceph 125 k 2024-04-25T09:20:53.475 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs x86_64 2:19.0.0-3194.ga126334d.el9 ceph 160 k 2024-04-25T09:20:53.475 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytest noarch 6.2.2-7.el9 appstream 519 k 2024-04-25T09:20:53.475 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados x86_64 2:19.0.0-3194.ga126334d.el9 ceph 315 k 2024-04-25T09:20:53.475 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd x86_64 2:19.0.0-3194.ga126334d.el9 ceph 293 k 2024-04-25T09:20:53.475 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw x86_64 2:19.0.0-3194.ga126334d.el9 ceph 99 k 2024-04-25T09:20:53.475 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse x86_64 2:19.0.0-3194.ga126334d.el9 ceph 90 k 2024-04-25T09:20:53.475 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror x86_64 2:19.0.0-3194.ga126334d.el9 ceph 3.4 M 2024-04-25T09:20:53.475 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd x86_64 2:19.0.0-3194.ga126334d.el9 ceph 175 k 2024-04-25T09:20:53.475 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2024-04-25T09:20:53.475 INFO:teuthology.orchestra.run.smithi164.stdout: librados2 x86_64 2:19.0.0-3194.ga126334d.el9 ceph 3.4 M 2024-04-25T09:20:53.476 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1 x86_64 2:19.0.0-3194.ga126334d.el9 ceph 3.7 M 2024-04-25T09:20:53.476 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-04-25T09:20:53.476 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T09:20:53.476 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T09:20:53.476 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common x86_64 2:19.0.0-3194.ga126334d.el9 ceph 19 M 2024-04-25T09:20:53.476 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 34 k 2024-04-25T09:20:53.476 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds x86_64 2:19.0.0-3194.ga126334d.el9 ceph 2.2 M 2024-04-25T09:20:53.476 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 249 k 2024-04-25T09:20:53.476 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon x86_64 2:19.0.0-3194.ga126334d.el9 ceph 5.0 M 2024-04-25T09:20:53.476 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd x86_64 2:19.0.0-3194.ga126334d.el9 ceph 71 M 2024-04-25T09:20:53.476 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 16 k 2024-04-25T09:20:53.477 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux x86_64 2:19.0.0-3194.ga126334d.el9 ceph 25 k 2024-04-25T09:20:53.477 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T09:20:53.477 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T09:20:53.477 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T09:20:53.477 INFO:teuthology.orchestra.run.smithi164.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T09:20:53.477 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T09:20:53.477 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T09:20:53.477 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T09:20:53.477 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T09:20:53.477 INFO:teuthology.orchestra.run.smithi164.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-25T09:20:53.477 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite x86_64 2:19.0.0-3194.ga126334d.el9 ceph 173 k 2024-04-25T09:20:53.478 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T09:20:53.478 INFO:teuthology.orchestra.run.smithi164.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T09:20:53.478 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T09:20:53.478 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T09:20:53.478 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T09:20:53.478 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 x86_64 2:19.0.0-3194.ga126334d.el9 ceph 548 k 2024-04-25T09:20:53.478 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T09:20:53.478 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2 x86_64 2:19.0.0-3194.ga126334d.el9 ceph 5.3 M 2024-04-25T09:20:53.478 INFO:teuthology.orchestra.run.smithi164.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-25T09:20:53.478 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T09:20:53.479 INFO:teuthology.orchestra.run.smithi164.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-04-25T09:20:53.479 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T09:20:53.479 INFO:teuthology.orchestra.run.smithi164.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T09:20:53.479 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T09:20:53.479 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T09:20:53.479 INFO:teuthology.orchestra.run.smithi164.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T09:20:53.479 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T09:20:53.479 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T09:20:53.479 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T09:20:53.479 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T09:20:53.479 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T09:20:53.480 INFO:teuthology.orchestra.run.smithi164.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T09:20:53.480 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T09:20:53.480 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T09:20:53.480 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T09:20:53.480 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse x86_64 2:19.0.0-3194.ga126334d.el9 ceph 45 k 2024-04-25T09:20:53.480 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common x86_64 2:19.0.0-3194.ga126334d.el9 ceph 135 k 2024-04-25T09:20:53.480 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T09:20:53.480 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T09:20:53.480 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T09:20:53.480 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T09:20:53.480 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T09:20:53.480 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T09:20:53.481 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T09:20:53.481 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T09:20:53.481 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T09:20:53.481 INFO:teuthology.orchestra.run.smithi164.stdout: python3-iniconfig noarch 1.1.1-7.el9 appstream 17 k 2024-04-25T09:20:53.481 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T09:20:53.481 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T09:20:53.481 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T09:20:53.481 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T09:20:53.481 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T09:20:53.481 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T09:20:53.481 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T09:20:53.481 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T09:20:53.482 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T09:20:53.482 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T09:20:53.482 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T09:20:53.482 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T09:20:53.482 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T09:20:53.482 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T09:20:53.482 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T09:20:53.482 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T09:20:53.482 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T09:20:53.482 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T09:20:53.482 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T09:20:53.483 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pluggy noarch 0.13.1-7.el9 appstream 41 k 2024-04-25T09:20:53.483 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T09:20:53.483 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T09:20:53.483 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T09:20:53.483 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T09:20:53.483 INFO:teuthology.orchestra.run.smithi164.stdout: python3-py noarch 1.10.0-6.el9 appstream 477 k 2024-04-25T09:20:53.483 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T09:20:53.483 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T09:20:53.483 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T09:20:53.483 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T09:20:53.483 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T09:20:53.483 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T09:20:53.483 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T09:20:53.484 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T09:20:53.484 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T09:20:53.484 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T09:20:53.484 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T09:20:53.484 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T09:20:53.484 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T09:20:53.484 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T09:20:53.484 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T09:20:53.484 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T09:20:53.484 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T09:20:53.484 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T09:20:53.484 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T09:20:53.485 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T09:20:53.485 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T09:20:53.485 INFO:teuthology.orchestra.run.smithi164.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T09:20:53.485 INFO:teuthology.orchestra.run.smithi164.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T09:20:53.485 INFO:teuthology.orchestra.run.smithi164.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T09:20:53.485 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T09:20:53.485 INFO:teuthology.orchestra.run.smithi164.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-25T09:20:53.485 INFO:teuthology.orchestra.run.smithi164.stdout:Installing weak dependencies: 2024-04-25T09:20:53.485 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T09:20:53.485 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:20:53.485 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-04-25T09:20:53.485 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================= 2024-04-25T09:20:53.486 INFO:teuthology.orchestra.run.smithi164.stdout:Install 133 Packages 2024-04-25T09:20:53.486 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 2 Packages 2024-04-25T09:20:53.486 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:20:53.486 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 292 M 2024-04-25T09:20:53.486 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-04-25T09:20:53.585 INFO:teuthology.orchestra.run.smithi043.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T09:20:53.586 INFO:teuthology.orchestra.run.smithi043.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T09:20:53.652 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T09:20:53.660 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================= 2024-04-25T09:20:53.660 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-25T09:20:53.660 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================= 2024-04-25T09:20:53.660 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T09:20:53.660 INFO:teuthology.orchestra.run.smithi043.stdout: ceph x86_64 2:19.0.0-3194.ga126334d.el9 ceph 6.5 k 2024-04-25T09:20:53.660 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base x86_64 2:19.0.0-3194.ga126334d.el9 ceph 5.9 M 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse x86_64 2:19.0.0-3194.ga126334d.el9 ceph 870 k 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3194.ga126334d.el9 ceph 157 k 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr x86_64 2:19.0.0-3194.ga126334d.el9 ceph 1.6 M 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 142 k 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 3.5 M 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 7.4 M 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 49 k 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw x86_64 2:19.0.0-3194.ga126334d.el9 ceph 10 M 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test x86_64 2:19.0.0-3194.ga126334d.el9 ceph 84 M 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 273 k 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 762 k 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel x86_64 2:19.0.0-3194.ga126334d.el9 ceph 32 k 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2 x86_64 2:19.0.0-3194.ga126334d.el9 ceph 796 k 2024-04-25T09:20:53.661 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel x86_64 2:19.0.0-3194.ga126334d.el9 ceph 125 k 2024-04-25T09:20:53.662 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs x86_64 2:19.0.0-3194.ga126334d.el9 ceph 160 k 2024-04-25T09:20:53.662 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytest noarch 6.2.2-7.el9 appstream 519 k 2024-04-25T09:20:53.662 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados x86_64 2:19.0.0-3194.ga126334d.el9 ceph 315 k 2024-04-25T09:20:53.662 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd x86_64 2:19.0.0-3194.ga126334d.el9 ceph 293 k 2024-04-25T09:20:53.662 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw x86_64 2:19.0.0-3194.ga126334d.el9 ceph 99 k 2024-04-25T09:20:53.662 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse x86_64 2:19.0.0-3194.ga126334d.el9 ceph 90 k 2024-04-25T09:20:53.662 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror x86_64 2:19.0.0-3194.ga126334d.el9 ceph 3.4 M 2024-04-25T09:20:53.662 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd x86_64 2:19.0.0-3194.ga126334d.el9 ceph 175 k 2024-04-25T09:20:53.662 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrading: 2024-04-25T09:20:53.662 INFO:teuthology.orchestra.run.smithi043.stdout: librados2 x86_64 2:19.0.0-3194.ga126334d.el9 ceph 3.4 M 2024-04-25T09:20:53.662 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1 x86_64 2:19.0.0-3194.ga126334d.el9 ceph 3.7 M 2024-04-25T09:20:53.662 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-25T09:20:53.662 INFO:teuthology.orchestra.run.smithi043.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common x86_64 2:19.0.0-3194.ga126334d.el9 ceph 19 M 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 34 k 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds x86_64 2:19.0.0-3194.ga126334d.el9 ceph 2.2 M 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 249 k 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon x86_64 2:19.0.0-3194.ga126334d.el9 ceph 5.0 M 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd x86_64 2:19.0.0-3194.ga126334d.el9 ceph 71 M 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3194.ga126334d.el9 ceph-noarch 16 k 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux x86_64 2:19.0.0-3194.ga126334d.el9 ceph 25 k 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T09:20:53.663 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite x86_64 2:19.0.0-3194.ga126334d.el9 ceph 173 k 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1 x86_64 2:19.0.0-3194.ga126334d.el9 ceph 548 k 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2 x86_64 2:19.0.0-3194.ga126334d.el9 ceph 5.3 M 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-25T09:20:53.664 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T09:20:53.665 INFO:teuthology.orchestra.run.smithi043.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-04-25T09:20:53.665 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T09:20:53.665 INFO:teuthology.orchestra.run.smithi043.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T09:20:53.665 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T09:20:53.665 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T09:20:53.665 INFO:teuthology.orchestra.run.smithi043.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T09:20:53.665 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T09:20:53.665 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T09:20:53.665 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T09:20:53.665 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T09:20:53.665 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T09:20:53.665 INFO:teuthology.orchestra.run.smithi043.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T09:20:53.665 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T09:20:53.666 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T09:20:53.666 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T09:20:53.666 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse x86_64 2:19.0.0-3194.ga126334d.el9 ceph 45 k 2024-04-25T09:20:53.666 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common x86_64 2:19.0.0-3194.ga126334d.el9 ceph 135 k 2024-04-25T09:20:53.666 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T09:20:53.666 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T09:20:53.666 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T09:20:53.666 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T09:20:53.666 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T09:20:53.666 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T09:20:53.666 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T09:20:53.666 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T09:20:53.666 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T09:20:53.667 INFO:teuthology.orchestra.run.smithi043.stdout: python3-iniconfig noarch 1.1.1-7.el9 appstream 17 k 2024-04-25T09:20:53.667 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T09:20:53.667 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T09:20:53.667 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T09:20:53.667 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T09:20:53.667 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T09:20:53.667 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T09:20:53.667 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T09:20:53.667 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T09:20:53.667 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T09:20:53.667 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T09:20:53.667 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T09:20:53.667 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pluggy noarch 0.13.1-7.el9 appstream 41 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-py noarch 1.10.0-6.el9 appstream 477 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T09:20:53.668 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T09:20:53.669 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T09:20:53.669 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T09:20:53.669 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T09:20:53.669 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T09:20:53.669 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T09:20:53.669 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T09:20:53.669 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T09:20:53.669 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T09:20:53.669 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T09:20:53.669 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T09:20:53.669 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T09:20:53.669 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T09:20:53.669 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout:Installing weak dependencies: 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T09:20:53.670 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:20:53.671 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T09:20:53.671 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================= 2024-04-25T09:20:53.671 INFO:teuthology.orchestra.run.smithi043.stdout:Install 133 Packages 2024-04-25T09:20:53.671 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrade 2 Packages 2024-04-25T09:20:53.671 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:20:53.671 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 292 M 2024-04-25T09:20:53.671 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T09:20:54.575 INFO:teuthology.orchestra.run.smithi164.stdout:(1/135): ceph-19.0.0-3194.ga126334d.el9.x86_64. 23 kB/s | 6.5 kB 00:00 2024-04-25T09:20:54.693 INFO:teuthology.orchestra.run.smithi043.stdout:(1/135): ceph-19.0.0-3194.ga126334d.el9.x86_64. 37 kB/s | 6.5 kB 00:00 2024-04-25T09:20:54.833 INFO:teuthology.orchestra.run.smithi164.stdout:(2/135): ceph-fuse-19.0.0-3194.ga126334d.el9.x8 3.3 MB/s | 870 kB 00:00 2024-04-25T09:20:54.942 INFO:teuthology.orchestra.run.smithi164.stdout:(3/135): ceph-immutable-object-cache-19.0.0-319 1.4 MB/s | 157 kB 00:00 2024-04-25T09:20:55.300 INFO:teuthology.orchestra.run.smithi164.stdout:(4/135): ceph-base-19.0.0-3194.ga126334d.el9.x8 5.8 MB/s | 5.9 MB 00:01 2024-04-25T09:20:55.486 INFO:teuthology.orchestra.run.smithi164.stdout:(5/135): ceph-mgr-19.0.0-3194.ga126334d.el9.x86 8.5 MB/s | 1.6 MB 00:00 2024-04-25T09:20:55.510 INFO:teuthology.orchestra.run.smithi043.stdout:(2/135): ceph-fuse-19.0.0-3194.ga126334d.el9.x8 1.0 MB/s | 870 kB 00:00 2024-04-25T09:20:55.609 INFO:teuthology.orchestra.run.smithi164.stdout:(6/135): ceph-mds-19.0.0-3194.ga126334d.el9.x86 3.3 MB/s | 2.2 MB 00:00 2024-04-25T09:20:55.685 INFO:teuthology.orchestra.run.smithi043.stdout:(3/135): ceph-immutable-object-cache-19.0.0-319 899 kB/s | 157 kB 00:00 2024-04-25T09:20:56.034 INFO:teuthology.orchestra.run.smithi164.stdout:(7/135): ceph-mon-19.0.0-3194.ga126334d.el9.x86 9.1 MB/s | 5.0 MB 00:00 2024-04-25T09:20:57.693 INFO:teuthology.orchestra.run.smithi164.stdout:(8/135): ceph-common-19.0.0-3194.ga126334d.el9. 5.7 MB/s | 19 MB 00:03 2024-04-25T09:20:57.826 INFO:teuthology.orchestra.run.smithi164.stdout:(9/135): ceph-radosgw-19.0.0-3194.ga126334d.el9 5.7 MB/s | 10 MB 00:01 2024-04-25T09:20:57.843 INFO:teuthology.orchestra.run.smithi043.stdout:(4/135): ceph-mds-19.0.0-3194.ga126334d.el9.x86 1.0 MB/s | 2.2 MB 00:02 2024-04-25T09:20:58.035 INFO:teuthology.orchestra.run.smithi164.stdout:(10/135): ceph-selinux-19.0.0-3194.ga126334d.el 73 kB/s | 25 kB 00:00 2024-04-25T09:20:58.110 INFO:teuthology.orchestra.run.smithi164.stdout:(11/135): libcephfs-devel-19.0.0-3194.ga126334d 423 kB/s | 32 kB 00:00 2024-04-25T09:20:58.394 INFO:teuthology.orchestra.run.smithi164.stdout:(12/135): libcephfs2-19.0.0-3194.ga126334d.el9. 2.7 MB/s | 796 kB 00:00 2024-04-25T09:20:58.452 INFO:teuthology.orchestra.run.smithi043.stdout:(5/135): ceph-common-19.0.0-3194.ga126334d.el9. 4.9 MB/s | 19 MB 00:03 2024-04-25T09:20:58.452 INFO:teuthology.orchestra.run.smithi164.stdout:(13/135): libcephsqlite-19.0.0-3194.ga126334d.e 2.9 MB/s | 173 kB 00:00 2024-04-25T09:20:58.536 INFO:teuthology.orchestra.run.smithi164.stdout:(14/135): librados-devel-19.0.0-3194.ga126334d. 1.5 MB/s | 125 kB 00:00 2024-04-25T09:20:58.745 INFO:teuthology.orchestra.run.smithi164.stdout:(15/135): libradosstriper1-19.0.0-3194.ga126334 2.6 MB/s | 548 kB 00:00 2024-04-25T09:20:59.027 INFO:teuthology.orchestra.run.smithi043.stdout:(6/135): ceph-mgr-19.0.0-3194.ga126334d.el9.x86 1.3 MB/s | 1.6 MB 00:01 2024-04-25T09:20:59.094 INFO:teuthology.orchestra.run.smithi043.stdout:(7/135): ceph-mon-19.0.0-3194.ga126334d.el9.x86 7.8 MB/s | 5.0 MB 00:00 2024-04-25T09:21:00.170 INFO:teuthology.orchestra.run.smithi164.stdout:(16/135): librgw2-19.0.0-3194.ga126334d.el9.x86 3.7 MB/s | 5.3 MB 00:01 2024-04-25T09:21:00.228 INFO:teuthology.orchestra.run.smithi164.stdout:(17/135): python3-ceph-argparse-19.0.0-3194.ga1 774 kB/s | 45 kB 00:00 2024-04-25T09:21:00.277 INFO:teuthology.orchestra.run.smithi043.stdout:(8/135): ceph-radosgw-19.0.0-3194.ga126334d.el9 8.6 MB/s | 10 MB 00:01 2024-04-25T09:21:00.287 INFO:teuthology.orchestra.run.smithi164.stdout:(18/135): python3-ceph-common-19.0.0-3194.ga126 2.3 MB/s | 135 kB 00:00 2024-04-25T09:21:00.346 INFO:teuthology.orchestra.run.smithi164.stdout:(19/135): python3-cephfs-19.0.0-3194.ga126334d. 2.7 MB/s | 160 kB 00:00 2024-04-25T09:21:00.361 INFO:teuthology.orchestra.run.smithi043.stdout:(9/135): ceph-base-19.0.0-3194.ga126334d.el9.x8 1.0 MB/s | 5.9 MB 00:05 2024-04-25T09:21:00.378 INFO:teuthology.orchestra.run.smithi043.stdout:(10/135): ceph-selinux-19.0.0-3194.ga126334d.el 251 kB/s | 25 kB 00:00 2024-04-25T09:21:00.436 INFO:teuthology.orchestra.run.smithi043.stdout:(11/135): libcephfs-devel-19.0.0-3194.ga126334d 543 kB/s | 32 kB 00:00 2024-04-25T09:21:00.446 INFO:teuthology.orchestra.run.smithi164.stdout:(20/135): python3-rados-19.0.0-3194.ga126334d.e 3.1 MB/s | 315 kB 00:00 2024-04-25T09:21:00.570 INFO:teuthology.orchestra.run.smithi043.stdout:(12/135): libcephfs2-19.0.0-3194.ga126334d.el9. 5.8 MB/s | 796 kB 00:00 2024-04-25T09:21:00.596 INFO:teuthology.orchestra.run.smithi164.stdout:(21/135): python3-rbd-19.0.0-3194.ga126334d.el9 1.9 MB/s | 293 kB 00:00 2024-04-25T09:21:00.629 INFO:teuthology.orchestra.run.smithi043.stdout:(13/135): libcephsqlite-19.0.0-3194.ga126334d.e 2.9 MB/s | 173 kB 00:00 2024-04-25T09:21:00.664 INFO:teuthology.orchestra.run.smithi164.stdout:(22/135): python3-rgw-19.0.0-3194.ga126334d.el9 1.4 MB/s | 99 kB 00:00 2024-04-25T09:21:00.679 INFO:teuthology.orchestra.run.smithi043.stdout:(14/135): librados-devel-19.0.0-3194.ga126334d. 2.4 MB/s | 125 kB 00:00 2024-04-25T09:21:00.730 INFO:teuthology.orchestra.run.smithi164.stdout:(23/135): rbd-fuse-19.0.0-3194.ga126334d.el9.x8 1.3 MB/s | 90 kB 00:00 2024-04-25T09:21:00.763 INFO:teuthology.orchestra.run.smithi043.stdout:(15/135): libradosstriper1-19.0.0-3194.ga126334 6.4 MB/s | 548 kB 00:00 2024-04-25T09:21:01.471 INFO:teuthology.orchestra.run.smithi043.stdout:(16/135): librgw2-19.0.0-3194.ga126334d.el9.x86 7.4 MB/s | 5.3 MB 00:00 2024-04-25T09:21:01.547 INFO:teuthology.orchestra.run.smithi043.stdout:(17/135): python3-ceph-argparse-19.0.0-3194.ga1 602 kB/s | 45 kB 00:00 2024-04-25T09:21:01.589 INFO:teuthology.orchestra.run.smithi164.stdout:(24/135): rbd-mirror-19.0.0-3194.ga126334d.el9. 4.0 MB/s | 3.4 MB 00:00 2024-04-25T09:21:01.647 INFO:teuthology.orchestra.run.smithi164.stdout:(25/135): rbd-nbd-19.0.0-3194.ga126334d.el9.x86 2.9 MB/s | 175 kB 00:00 2024-04-25T09:21:01.714 INFO:teuthology.orchestra.run.smithi164.stdout:(26/135): ceph-grafana-dashboards-19.0.0-3194.g 512 kB/s | 34 kB 00:00 2024-04-25T09:21:01.781 INFO:teuthology.orchestra.run.smithi164.stdout:(27/135): ceph-mgr-cephadm-19.0.0-3194.ga126334 2.1 MB/s | 142 kB 00:00 2024-04-25T09:21:01.872 INFO:teuthology.orchestra.run.smithi043.stdout:(18/135): python3-ceph-common-19.0.0-3194.ga126 414 kB/s | 135 kB 00:00 2024-04-25T09:21:01.947 INFO:teuthology.orchestra.run.smithi043.stdout:(19/135): python3-cephfs-19.0.0-3194.ga126334d. 2.1 MB/s | 160 kB 00:00 2024-04-25T09:21:02.031 INFO:teuthology.orchestra.run.smithi043.stdout:(20/135): python3-rados-19.0.0-3194.ga126334d.e 3.7 MB/s | 315 kB 00:00 2024-04-25T09:21:02.115 INFO:teuthology.orchestra.run.smithi043.stdout:(21/135): python3-rbd-19.0.0-3194.ga126334d.el9 3.4 MB/s | 293 kB 00:00 2024-04-25T09:21:02.173 INFO:teuthology.orchestra.run.smithi043.stdout:(22/135): python3-rgw-19.0.0-3194.ga126334d.el9 1.7 MB/s | 99 kB 00:00 2024-04-25T09:21:02.241 INFO:teuthology.orchestra.run.smithi043.stdout:(23/135): rbd-fuse-19.0.0-3194.ga126334d.el9.x8 1.3 MB/s | 90 kB 00:00 2024-04-25T09:21:02.665 INFO:teuthology.orchestra.run.smithi164.stdout:(28/135): ceph-mgr-dashboard-19.0.0-3194.ga1263 4.0 MB/s | 3.5 MB 00:00 2024-04-25T09:21:02.724 INFO:teuthology.orchestra.run.smithi043.stdout:(24/135): rbd-mirror-19.0.0-3194.ga126334d.el9. 7.0 MB/s | 3.4 MB 00:00 2024-04-25T09:21:02.808 INFO:teuthology.orchestra.run.smithi043.stdout:(25/135): rbd-nbd-19.0.0-3194.ga126334d.el9.x86 2.1 MB/s | 175 kB 00:00 2024-04-25T09:21:02.883 INFO:teuthology.orchestra.run.smithi043.stdout:(26/135): ceph-grafana-dashboards-19.0.0-3194.g 456 kB/s | 34 kB 00:00 2024-04-25T09:21:02.950 INFO:teuthology.orchestra.run.smithi043.stdout:(27/135): ceph-mgr-cephadm-19.0.0-3194.ga126334 2.1 MB/s | 142 kB 00:00 2024-04-25T09:21:03.467 INFO:teuthology.orchestra.run.smithi043.stdout:(28/135): ceph-mgr-dashboard-19.0.0-3194.ga1263 6.8 MB/s | 3.5 MB 00:00 2024-04-25T09:21:04.467 INFO:teuthology.orchestra.run.smithi043.stdout:(29/135): ceph-mgr-diskprediction-local-19.0.0- 7.4 MB/s | 7.4 MB 00:00 2024-04-25T09:21:04.542 INFO:teuthology.orchestra.run.smithi043.stdout:(30/135): ceph-mgr-modules-core-19.0.0-3194.ga1 3.2 MB/s | 249 kB 00:00 2024-04-25T09:21:04.573 INFO:teuthology.orchestra.run.smithi164.stdout:(29/135): ceph-mgr-diskprediction-local-19.0.0- 3.9 MB/s | 7.4 MB 00:01 2024-04-25T09:21:04.609 INFO:teuthology.orchestra.run.smithi043.stdout:(31/135): ceph-mgr-rook-19.0.0-3194.ga126334d.e 742 kB/s | 49 kB 00:00 2024-04-25T09:21:04.657 INFO:teuthology.orchestra.run.smithi164.stdout:(30/135): ceph-mgr-modules-core-19.0.0-3194.ga1 2.9 MB/s | 249 kB 00:00 2024-04-25T09:21:04.684 INFO:teuthology.orchestra.run.smithi043.stdout:(32/135): ceph-prometheus-alerts-19.0.0-3194.ga 219 kB/s | 16 kB 00:00 2024-04-25T09:21:04.732 INFO:teuthology.orchestra.run.smithi164.stdout:(31/135): ceph-mgr-rook-19.0.0-3194.ga126334d.e 656 kB/s | 49 kB 00:00 2024-04-25T09:21:04.760 INFO:teuthology.orchestra.run.smithi043.stdout:(33/135): ceph-volume-19.0.0-3194.ga126334d.el9 3.6 MB/s | 273 kB 00:00 2024-04-25T09:21:04.807 INFO:teuthology.orchestra.run.smithi164.stdout:(32/135): ceph-prometheus-alerts-19.0.0-3194.ga 219 kB/s | 16 kB 00:00 2024-04-25T09:21:04.910 INFO:teuthology.orchestra.run.smithi164.stdout:(33/135): ceph-volume-19.0.0-3194.ga126334d.el9 2.6 MB/s | 273 kB 00:00 2024-04-25T09:21:04.910 INFO:teuthology.orchestra.run.smithi043.stdout:(34/135): cephadm-19.0.0-3194.ga126334d.el9.noa 5.0 MB/s | 762 kB 00:00 2024-04-25T09:21:05.044 INFO:teuthology.orchestra.run.smithi043.stdout:(35/135): fuse-2.9.9-15.el9.x86_64.rpm 602 kB/s | 80 kB 00:00 2024-04-25T09:21:05.102 INFO:teuthology.orchestra.run.smithi164.stdout:(34/135): cephadm-19.0.0-3194.ga126334d.el9.noa 3.9 MB/s | 762 kB 00:00 2024-04-25T09:21:05.218 INFO:teuthology.orchestra.run.smithi164.stdout:(35/135): fuse-2.9.9-15.el9.x86_64.rpm 690 kB/s | 80 kB 00:00 2024-04-25T09:21:05.227 INFO:teuthology.orchestra.run.smithi043.stdout:(36/135): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-04-25T09:21:05.278 INFO:teuthology.orchestra.run.smithi043.stdout:(37/135): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-04-25T09:21:05.328 INFO:teuthology.orchestra.run.smithi043.stdout:(38/135): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-25T09:21:05.344 INFO:teuthology.orchestra.run.smithi164.stdout:(36/135): libgfortran-11.4.1-3.el9.x86_64.rpm 6.3 MB/s | 803 kB 00:00 2024-04-25T09:21:05.394 INFO:teuthology.orchestra.run.smithi164.stdout:(37/135): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-04-25T09:21:05.395 INFO:teuthology.orchestra.run.smithi043.stdout:(39/135): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-04-25T09:21:05.444 INFO:teuthology.orchestra.run.smithi164.stdout:(38/135): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-25T09:21:05.494 INFO:teuthology.orchestra.run.smithi164.stdout:(39/135): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-25T09:21:05.570 INFO:teuthology.orchestra.run.smithi043.stdout:(40/135): python3-cryptography-36.0.1-4.el9.x86 7.0 MB/s | 1.2 MB 00:00 2024-04-25T09:21:05.578 INFO:teuthology.orchestra.run.smithi164.stdout:(40/135): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.2 MB 00:00 2024-04-25T09:21:05.620 INFO:teuthology.orchestra.run.smithi043.stdout:(41/135): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-25T09:21:05.628 INFO:teuthology.orchestra.run.smithi164.stdout:(41/135): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-25T09:21:05.671 INFO:teuthology.orchestra.run.smithi043.stdout:(42/135): python3-pycparser-2.20-6.el9.noarch.r 2.7 MB/s | 135 kB 00:00 2024-04-25T09:21:05.678 INFO:teuthology.orchestra.run.smithi164.stdout:(42/135): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-25T09:21:05.721 INFO:teuthology.orchestra.run.smithi043.stdout:(43/135): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-25T09:21:05.729 INFO:teuthology.orchestra.run.smithi164.stdout:(43/135): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-25T09:21:05.771 INFO:teuthology.orchestra.run.smithi043.stdout:(44/135): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T09:21:05.779 INFO:teuthology.orchestra.run.smithi164.stdout:(44/135): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T09:21:05.996 INFO:teuthology.orchestra.run.smithi164.stdout:(45/135): boost-program-options-1.75.0-8.el9.x8 493 kB/s | 107 kB 00:00 2024-04-25T09:21:05.996 INFO:teuthology.orchestra.run.smithi043.stdout:(45/135): boost-program-options-1.75.0-8.el9.x8 475 kB/s | 107 kB 00:00 2024-04-25T09:21:06.055 INFO:teuthology.orchestra.run.smithi164.stdout:(46/135): flexiblas-3.0.4-8.el9.x86_64.rpm 560 kB/s | 33 kB 00:00 2024-04-25T09:21:06.057 INFO:teuthology.orchestra.run.smithi043.stdout:(46/135): flexiblas-3.0.4-8.el9.x86_64.rpm 541 kB/s | 33 kB 00:00 2024-04-25T09:21:06.797 INFO:teuthology.orchestra.run.smithi164.stdout:(47/135): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.0 MB/s | 3.0 MB 00:00 2024-04-25T09:21:06.856 INFO:teuthology.orchestra.run.smithi164.stdout:(48/135): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-04-25T09:21:06.933 INFO:teuthology.orchestra.run.smithi043.stdout:(47/135): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.4 MB/s | 3.0 MB 00:00 2024-04-25T09:21:06.972 INFO:teuthology.orchestra.run.smithi164.stdout:(49/135): libasan8-13.2.1-6.3.el9.x86_64.rpm 3.9 MB/s | 469 kB 00:00 2024-04-25T09:21:06.991 INFO:teuthology.orchestra.run.smithi043.stdout:(48/135): flexiblas-openblas-openmp-3.0.4-8.el9 314 kB/s | 18 kB 00:00 2024-04-25T09:21:07.031 INFO:teuthology.orchestra.run.smithi164.stdout:(50/135): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-25T09:21:07.098 INFO:teuthology.orchestra.run.smithi164.stdout:(51/135): librabbitmq-0.11.0-7.el9.x86_64.rpm 680 kB/s | 45 kB 00:00 2024-04-25T09:21:07.116 INFO:teuthology.orchestra.run.smithi043.stdout:(49/135): libasan8-13.2.1-6.3.el9.x86_64.rpm 3.7 MB/s | 469 kB 00:00 2024-04-25T09:21:07.208 INFO:teuthology.orchestra.run.smithi043.stdout:(50/135): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-04-25T09:21:07.275 INFO:teuthology.orchestra.run.smithi043.stdout:(51/135): librabbitmq-0.11.0-7.el9.x86_64.rpm 681 kB/s | 45 kB 00:00 2024-04-25T09:21:07.289 INFO:teuthology.orchestra.run.smithi164.stdout:(52/135): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-04-25T09:21:07.374 INFO:teuthology.orchestra.run.smithi164.stdout:(53/135): libubsan-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-04-25T09:21:07.449 INFO:teuthology.orchestra.run.smithi164.stdout:(54/135): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-25T09:21:07.459 INFO:teuthology.orchestra.run.smithi043.stdout:(52/135): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 662 kB 00:00 2024-04-25T09:21:07.515 INFO:teuthology.orchestra.run.smithi164.stdout:(55/135): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-04-25T09:21:07.518 INFO:teuthology.orchestra.run.smithi043.stdout:(53/135): libubsan-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-04-25T09:21:07.574 INFO:teuthology.orchestra.run.smithi164.stdout:(56/135): openblas-0.3.21-2.el9.x86_64.rpm 603 kB/s | 35 kB 00:00 2024-04-25T09:21:07.634 INFO:teuthology.orchestra.run.smithi043.stdout:(54/135): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 292 kB 00:00 2024-04-25T09:21:07.701 INFO:teuthology.orchestra.run.smithi043.stdout:(55/135): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-04-25T09:21:07.760 INFO:teuthology.orchestra.run.smithi043.stdout:(56/135): openblas-0.3.21-2.el9.x86_64.rpm 604 kB/s | 35 kB 00:00 2024-04-25T09:21:08.050 INFO:teuthology.orchestra.run.smithi164.stdout:(57/135): openblas-openmp-0.3.21-2.el9.x86_64.r 10 MB/s | 4.7 MB 00:00 2024-04-25T09:21:08.158 INFO:teuthology.orchestra.run.smithi164.stdout:(58/135): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-04-25T09:21:08.233 INFO:teuthology.orchestra.run.smithi164.stdout:(59/135): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-04-25T09:21:08.260 INFO:teuthology.orchestra.run.smithi043.stdout:(57/135): openblas-openmp-0.3.21-2.el9.x86_64.r 9.5 MB/s | 4.7 MB 00:00 2024-04-25T09:21:08.402 INFO:teuthology.orchestra.run.smithi043.stdout:(58/135): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-04-25T09:21:08.503 INFO:teuthology.orchestra.run.smithi043.stdout:(59/135): python3-attrs-20.3.0-7.el9.noarch.rpm 874 kB/s | 87 kB 00:00 2024-04-25T09:21:08.758 INFO:teuthology.orchestra.run.smithi164.stdout:(60/135): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-04-25T09:21:08.817 INFO:teuthology.orchestra.run.smithi164.stdout:(61/135): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-25T09:21:08.876 INFO:teuthology.orchestra.run.smithi164.stdout:(62/135): python3-iniconfig-1.1.1-7.el9.noarch. 299 kB/s | 17 kB 00:00 2024-04-25T09:21:08.920 INFO:teuthology.orchestra.run.smithi043.stdout:(60/135): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-04-25T09:21:08.934 INFO:teuthology.orchestra.run.smithi164.stdout:(63/135): python3-jinja2-2.11.3-5.el9.noarch.rp 4.1 MB/s | 248 kB 00:00 2024-04-25T09:21:08.993 INFO:teuthology.orchestra.run.smithi164.stdout:(64/135): python3-jsonpatch-1.21-16.el9.noarch. 440 kB/s | 26 kB 00:00 2024-04-25T09:21:08.995 INFO:teuthology.orchestra.run.smithi043.stdout:(61/135): python3-devel-3.9.18-3.el9.x86_64.rpm 3.2 MB/s | 244 kB 00:00 2024-04-25T09:21:09.060 INFO:teuthology.orchestra.run.smithi164.stdout:(65/135): python3-jsonpointer-2.0-4.el9.noarch. 287 kB/s | 19 kB 00:00 2024-04-25T09:21:09.062 INFO:teuthology.orchestra.run.smithi043.stdout:(62/135): python3-iniconfig-1.1.1-7.el9.noarch. 262 kB/s | 17 kB 00:00 2024-04-25T09:21:09.146 INFO:teuthology.orchestra.run.smithi043.stdout:(63/135): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-04-25T09:21:09.204 INFO:teuthology.orchestra.run.smithi043.stdout:(64/135): python3-jsonpatch-1.21-16.el9.noarch. 442 kB/s | 26 kB 00:00 2024-04-25T09:21:09.262 INFO:teuthology.orchestra.run.smithi043.stdout:(65/135): python3-jsonpointer-2.0-4.el9.noarch. 328 kB/s | 19 kB 00:00 2024-04-25T09:21:09.321 INFO:teuthology.orchestra.run.smithi043.stdout:(66/135): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-25T09:21:09.380 INFO:teuthology.orchestra.run.smithi043.stdout:(67/135): python3-markupsafe-1.1.1-12.el9.x86_6 598 kB/s | 35 kB 00:00 2024-04-25T09:21:09.747 INFO:teuthology.orchestra.run.smithi043.stdout:(68/135): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-04-25T09:21:09.752 INFO:teuthology.orchestra.run.smithi164.stdout:(66/135): ceph-test-19.0.0-3194.ga126334d.el9.x 7.1 MB/s | 84 MB 00:11 2024-04-25T09:21:09.777 INFO:teuthology.orchestra.run.smithi164.stdout:(67/135): python3-mako-1.1.4-6.el9.noarch.rpm 240 kB/s | 172 kB 00:00 2024-04-25T09:21:09.855 INFO:teuthology.orchestra.run.smithi043.stdout:(69/135): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-25T09:21:09.911 INFO:teuthology.orchestra.run.smithi164.stdout:(68/135): python3-markupsafe-1.1.1-12.el9.x86_6 220 kB/s | 35 kB 00:00 2024-04-25T09:21:09.931 INFO:teuthology.orchestra.run.smithi043.stdout:(70/135): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-04-25T09:21:10.006 INFO:teuthology.orchestra.run.smithi043.stdout:(71/135): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 77 kB 00:00 2024-04-25T09:21:10.073 INFO:teuthology.orchestra.run.smithi043.stdout:(72/135): python3-pluggy-0.13.1-7.el9.noarch.rp 621 kB/s | 41 kB 00:00 2024-04-25T09:21:10.140 INFO:teuthology.orchestra.run.smithi043.stdout:(73/135): python3-prettytable-0.7.2-27.el9.noar 633 kB/s | 42 kB 00:00 2024-04-25T09:21:10.219 INFO:teuthology.orchestra.run.smithi164.stdout:(69/135): python3-numpy-f2py-1.20.1-5.el9.x86_6 813 kB/s | 250 kB 00:00 2024-04-25T09:21:10.240 INFO:teuthology.orchestra.run.smithi043.stdout:(74/135): python3-protobuf-3.14.0-13.el9.noarch 2.6 MB/s | 269 kB 00:00 2024-04-25T09:21:10.295 INFO:teuthology.orchestra.run.smithi164.stdout:(70/135): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.9 MB/s | 5.1 MB 00:00 2024-04-25T09:21:10.324 INFO:teuthology.orchestra.run.smithi043.stdout:(75/135): python3-py-1.10.0-6.el9.noarch.rpm 5.6 MB/s | 477 kB 00:00 2024-04-25T09:21:10.362 INFO:teuthology.orchestra.run.smithi164.stdout:(71/135): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-04-25T09:21:10.391 INFO:teuthology.orchestra.run.smithi043.stdout:(76/135): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-04-25T09:21:10.412 INFO:teuthology.orchestra.run.smithi164.stdout:(72/135): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 222 kB 00:00 2024-04-25T09:21:10.437 INFO:teuthology.orchestra.run.smithi164.stdout:(73/135): python3-pluggy-0.13.1-7.el9.noarch.rp 550 kB/s | 41 kB 00:00 2024-04-25T09:21:10.466 INFO:teuthology.orchestra.run.smithi043.stdout:(77/135): python3-pyasn1-modules-0.4.8-6.el9.no 3.7 MB/s | 279 kB 00:00 2024-04-25T09:21:10.479 INFO:teuthology.orchestra.run.smithi164.stdout:(74/135): python3-prettytable-0.7.2-27.el9.noar 632 kB/s | 42 kB 00:00 2024-04-25T09:21:10.513 INFO:teuthology.orchestra.run.smithi164.stdout:(75/135): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-04-25T09:21:10.533 INFO:teuthology.orchestra.run.smithi043.stdout:(78/135): python3-pytest-6.2.2-7.el9.noarch.rpm 7.6 MB/s | 519 kB 00:00 2024-04-25T09:21:10.571 INFO:teuthology.orchestra.run.smithi164.stdout:(76/135): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-25T09:21:10.600 INFO:teuthology.orchestra.run.smithi043.stdout:(79/135): python3-pytz-2021.1-5.el9.noarch.rpm 766 kB/s | 51 kB 00:00 2024-04-25T09:21:10.638 INFO:teuthology.orchestra.run.smithi164.stdout:(77/135): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-04-25T09:21:10.659 INFO:teuthology.orchestra.run.smithi043.stdout:(80/135): python3-requests-oauthlib-1.3.0-12.el 925 kB/s | 54 kB 00:00 2024-04-25T09:21:10.714 INFO:teuthology.orchestra.run.smithi164.stdout:(78/135): python3-pytest-6.2.2-7.el9.noarch.rpm 6.8 MB/s | 519 kB 00:00 2024-04-25T09:21:10.739 INFO:teuthology.orchestra.run.smithi164.stdout:(79/135): python3-py-1.10.0-6.el9.noarch.rpm 1.8 MB/s | 477 kB 00:00 2024-04-25T09:21:10.789 INFO:teuthology.orchestra.run.smithi164.stdout:(80/135): python3-pytz-2021.1-5.el9.noarch.rpm 679 kB/s | 51 kB 00:00 2024-04-25T09:21:10.814 INFO:teuthology.orchestra.run.smithi164.stdout:(81/135): python3-requests-oauthlib-1.3.0-12.el 714 kB/s | 54 kB 00:00 2024-04-25T09:21:10.898 INFO:teuthology.orchestra.run.smithi164.stdout:(82/135): python3-toml-0.10.2-6.el9.noarch.rpm 501 kB/s | 42 kB 00:00 2024-04-25T09:21:11.040 INFO:teuthology.orchestra.run.smithi164.stdout:(83/135): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-04-25T09:21:11.124 INFO:teuthology.orchestra.run.smithi164.stdout:(84/135): qatlib-service-24.02.0-1.el9.x86_64.r 431 kB/s | 36 kB 00:00 2024-04-25T09:21:11.207 INFO:teuthology.orchestra.run.smithi164.stdout:(85/135): qatzip-libs-1.2.0-1.el9.x86_64.rpm 570 kB/s | 47 kB 00:00 2024-04-25T09:21:11.341 INFO:teuthology.orchestra.run.smithi164.stdout:(86/135): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 305 kB 00:00 2024-04-25T09:21:11.400 INFO:teuthology.orchestra.run.smithi164.stdout:(87/135): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-25T09:21:11.434 INFO:teuthology.orchestra.run.smithi043.stdout:(81/135): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-04-25T09:21:11.449 INFO:teuthology.orchestra.run.smithi164.stdout:(88/135): lua-devel-5.4.4-4.el9.x86_64.rpm 447 kB/s | 22 kB 00:00 2024-04-25T09:21:11.521 INFO:teuthology.orchestra.run.smithi043.stdout:(82/135): python3-toml-0.10.2-6.el9.noarch.rpm 480 kB/s | 42 kB 00:00 2024-04-25T09:21:11.585 INFO:teuthology.orchestra.run.smithi043.stdout:(83/135): qatlib-24.02.0-1.el9.x86_64.rpm 3.4 MB/s | 221 kB 00:00 2024-04-25T09:21:11.641 INFO:teuthology.orchestra.run.smithi164.stdout:(89/135): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-04-25T09:21:11.651 INFO:teuthology.orchestra.run.smithi043.stdout:(84/135): qatlib-service-24.02.0-1.el9.x86_64.r 540 kB/s | 36 kB 00:00 2024-04-25T09:21:11.710 INFO:teuthology.orchestra.run.smithi043.stdout:(85/135): qatzip-libs-1.2.0-1.el9.x86_64.rpm 817 kB/s | 47 kB 00:00 2024-04-25T09:21:11.785 INFO:teuthology.orchestra.run.smithi043.stdout:(86/135): socat-1.7.4.1-5.el9.x86_64.rpm 4.0 MB/s | 305 kB 00:00 2024-04-25T09:21:11.792 INFO:teuthology.orchestra.run.smithi164.stdout:(90/135): protobuf-compiler-3.14.0-13.el9.x86_6 2.5 MB/s | 863 kB 00:00 2024-04-25T09:21:11.844 INFO:teuthology.orchestra.run.smithi043.stdout:(87/135): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-25T09:21:11.928 INFO:teuthology.orchestra.run.smithi043.stdout:(88/135): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-04-25T09:21:11.975 INFO:teuthology.orchestra.run.smithi164.stdout:(91/135): gperftools-libs-2.9.1-2.el9.x86_64.rp 1.6 MB/s | 309 kB 00:00 2024-04-25T09:21:12.018 INFO:teuthology.orchestra.run.smithi164.stdout:(92/135): grpc-data-1.46.7-10.el9.noarch.rpm 464 kB/s | 19 kB 00:00 2024-04-25T09:21:12.043 INFO:teuthology.orchestra.run.smithi164.stdout:(93/135): abseil-cpp-20211102.0-3.el9.x86_64.rp 1.3 MB/s | 548 kB 00:00 2024-04-25T09:21:12.069 INFO:teuthology.orchestra.run.smithi043.stdout:(89/135): protobuf-compiler-3.14.0-13.el9.x86_6 6.0 MB/s | 863 kB 00:00 2024-04-25T09:21:12.093 INFO:teuthology.orchestra.run.smithi164.stdout:(94/135): libarrow-doc-9.0.0-10.el9.noarch.rpm 519 kB/s | 26 kB 00:00 2024-04-25T09:21:12.152 INFO:teuthology.orchestra.run.smithi164.stdout:(95/135): liboath-2.6.7-2.el9.x86_64.rpm 834 kB/s | 49 kB 00:00 2024-04-25T09:21:12.218 INFO:teuthology.orchestra.run.smithi164.stdout:(96/135): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-25T09:21:12.344 INFO:teuthology.orchestra.run.smithi164.stdout:(97/135): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-25T09:21:12.486 INFO:teuthology.orchestra.run.smithi043.stdout:(90/135): abseil-cpp-20211102.0-3.el9.x86_64.rp 1.3 MB/s | 548 kB 00:00 2024-04-25T09:21:12.570 INFO:teuthology.orchestra.run.smithi043.stdout:(91/135): gperftools-libs-2.9.1-2.el9.x86_64.rp 3.6 MB/s | 309 kB 00:00 2024-04-25T09:21:12.636 INFO:teuthology.orchestra.run.smithi164.stdout:(98/135): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 838 kB 00:00 2024-04-25T09:21:12.670 INFO:teuthology.orchestra.run.smithi043.stdout:(92/135): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-04-25T09:21:12.911 INFO:teuthology.orchestra.run.smithi164.stdout:(99/135): python3-asyncssh-2.13.2-5.el9.noarch. 1.9 MB/s | 548 kB 00:00 2024-04-25T09:21:12.946 INFO:teuthology.orchestra.run.smithi043.stdout:(93/135): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-25T09:21:12.961 INFO:teuthology.orchestra.run.smithi164.stdout:(100/135): python3-bcrypt-3.2.2-1.el9.x86_64.rp 872 kB/s | 43 kB 00:00 2024-04-25T09:21:13.028 INFO:teuthology.orchestra.run.smithi164.stdout:(101/135): python3-cachetools-4.2.4-1.el9.noarc 483 kB/s | 32 kB 00:00 2024-04-25T09:21:13.029 INFO:teuthology.orchestra.run.smithi043.stdout:(94/135): libarrow-doc-9.0.0-10.el9.noarch.rpm 313 kB/s | 26 kB 00:00 2024-04-25T09:21:13.070 INFO:teuthology.orchestra.run.smithi164.stdout:(102/135): python3-certifi-2021.10.8-2.el9.noar 362 kB/s | 15 kB 00:00 2024-04-25T09:21:13.096 INFO:teuthology.orchestra.run.smithi043.stdout:(95/135): liboath-2.6.7-2.el9.x86_64.rpm 729 kB/s | 49 kB 00:00 2024-04-25T09:21:13.155 INFO:teuthology.orchestra.run.smithi043.stdout:(96/135): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-25T09:21:13.170 INFO:teuthology.orchestra.run.smithi164.stdout:(103/135): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-25T09:21:13.221 INFO:teuthology.orchestra.run.smithi043.stdout:(97/135): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-25T09:21:13.305 INFO:teuthology.orchestra.run.smithi043.stdout:(98/135): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-04-25T09:21:13.414 INFO:teuthology.orchestra.run.smithi043.stdout:(99/135): python3-asyncssh-2.13.2-5.el9.noarch. 4.9 MB/s | 548 kB 00:00 2024-04-25T09:21:13.481 INFO:teuthology.orchestra.run.smithi043.stdout:(100/135): python3-bcrypt-3.2.2-1.el9.x86_64.rp 652 kB/s | 43 kB 00:00 2024-04-25T09:21:13.714 INFO:teuthology.orchestra.run.smithi043.stdout:(101/135): python3-cachetools-4.2.4-1.el9.noarc 138 kB/s | 32 kB 00:00 2024-04-25T09:21:13.831 INFO:teuthology.orchestra.run.smithi043.stdout:(102/135): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-04-25T09:21:13.907 INFO:teuthology.orchestra.run.smithi043.stdout:(103/135): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-25T09:21:13.979 INFO:teuthology.orchestra.run.smithi164.stdout:(104/135): ceph-osd-19.0.0-3194.ga126334d.el9.x 3.9 MB/s | 71 MB 00:18 2024-04-25T09:21:13.982 INFO:teuthology.orchestra.run.smithi043.stdout:(104/135): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-25T09:21:14.099 INFO:teuthology.orchestra.run.smithi043.stdout:(105/135): python3-google-auth-2.29.0-1.el9.noa 1.7 MB/s | 206 kB 00:00 2024-04-25T09:21:14.212 INFO:teuthology.orchestra.run.smithi164.stdout:(105/135): python3-google-auth-2.29.0-1.el9.noa 883 kB/s | 206 kB 00:00 2024-04-25T09:21:14.333 INFO:teuthology.orchestra.run.smithi043.stdout:(106/135): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-04-25T09:21:14.416 INFO:teuthology.orchestra.run.smithi043.stdout:(107/135): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-04-25T09:21:14.474 INFO:teuthology.orchestra.run.smithi043.stdout:(108/135): python3-jaraco-8.2.1-3.el9.noarch.rp 184 kB/s | 11 kB 00:00 2024-04-25T09:21:14.533 INFO:teuthology.orchestra.run.smithi043.stdout:(109/135): python3-jaraco-classes-3.2.1-5.el9.n 303 kB/s | 18 kB 00:00 2024-04-25T09:21:14.546 INFO:teuthology.orchestra.run.smithi164.stdout:(106/135): python3-cherrypy-18.6.1-2.el9.noarch 261 kB/s | 358 kB 00:01 2024-04-25T09:21:14.609 INFO:teuthology.orchestra.run.smithi043.stdout:(110/135): python3-jaraco-collections-3.0.0-8.e 308 kB/s | 23 kB 00:00 2024-04-25T09:21:14.621 INFO:teuthology.orchestra.run.smithi164.stdout:(107/135): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-25T09:21:14.663 INFO:teuthology.orchestra.run.smithi164.stdout:(108/135): python3-jaraco-8.2.1-3.el9.noarch.rp 258 kB/s | 11 kB 00:00 2024-04-25T09:21:14.675 INFO:teuthology.orchestra.run.smithi043.stdout:(111/135): python3-jaraco-functools-3.5.0-2.el9 292 kB/s | 19 kB 00:00 2024-04-25T09:21:14.721 INFO:teuthology.orchestra.run.smithi164.stdout:(109/135): python3-grpcio-1.46.7-10.el9.x86_64. 4.0 MB/s | 2.0 MB 00:00 2024-04-25T09:21:14.742 INFO:teuthology.orchestra.run.smithi043.stdout:(112/135): python3-jaraco-text-3.2.0-6.el9.noar 296 kB/s | 20 kB 00:00 2024-04-25T09:21:14.747 INFO:teuthology.orchestra.run.smithi164.stdout:(110/135): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-04-25T09:21:14.772 INFO:teuthology.orchestra.run.smithi164.stdout:(111/135): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-04-25T09:21:14.797 INFO:teuthology.orchestra.run.smithi164.stdout:(112/135): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-04-25T09:21:14.823 INFO:teuthology.orchestra.run.smithi164.stdout:(113/135): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-04-25T09:21:14.873 INFO:teuthology.orchestra.run.smithi164.stdout:(114/135): python3-logutils-0.3.5-21.el9.noarch 929 kB/s | 46 kB 00:00 2024-04-25T09:21:14.917 INFO:teuthology.orchestra.run.smithi043.stdout:(113/135): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-04-25T09:21:14.923 INFO:teuthology.orchestra.run.smithi164.stdout:(115/135): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-25T09:21:14.973 INFO:teuthology.orchestra.run.smithi164.stdout:(116/135): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-04-25T09:21:14.993 INFO:teuthology.orchestra.run.smithi043.stdout:(114/135): python3-logutils-0.3.5-21.el9.noarch 619 kB/s | 46 kB 00:00 2024-04-25T09:21:15.040 INFO:teuthology.orchestra.run.smithi164.stdout:(117/135): libarrow-9.0.0-10.el9.x86_64.rpm 1.5 MB/s | 4.4 MB 00:03 2024-04-25T09:21:15.060 INFO:teuthology.orchestra.run.smithi043.stdout:(115/135): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-04-25T09:21:15.099 INFO:teuthology.orchestra.run.smithi164.stdout:(118/135): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-04-25T09:21:15.135 INFO:teuthology.orchestra.run.smithi043.stdout:(116/135): python3-natsort-7.1.1-5.el9.noarch.r 771 kB/s | 58 kB 00:00 2024-04-25T09:21:15.149 INFO:teuthology.orchestra.run.smithi164.stdout:(119/135): python3-kubernetes-26.1.0-2.el9.noar 2.9 MB/s | 1.0 MB 00:00 2024-04-25T09:21:15.174 INFO:teuthology.orchestra.run.smithi164.stdout:(120/135): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-25T09:21:15.199 INFO:teuthology.orchestra.run.smithi164.stdout:(121/135): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-04-25T09:21:15.210 INFO:teuthology.orchestra.run.smithi043.stdout:(117/135): python3-pecan-1.4.2-3.el9.noarch.rpm 3.6 MB/s | 272 kB 00:00 2024-04-25T09:21:15.242 INFO:teuthology.orchestra.run.smithi164.stdout:(122/135): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-04-25T09:21:15.267 INFO:teuthology.orchestra.run.smithi164.stdout:(123/135): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-04-25T09:21:15.277 INFO:teuthology.orchestra.run.smithi043.stdout:(118/135): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-04-25T09:21:15.292 INFO:teuthology.orchestra.run.smithi164.stdout:(124/135): python3-portend-3.1.0-2.el9.noarch.r 65 kB/s | 16 kB 00:00 2024-04-25T09:21:15.318 INFO:teuthology.orchestra.run.smithi164.stdout:(125/135): python3-tempora-5.0.0-2.el9.noarch.r 472 kB/s | 36 kB 00:00 2024-04-25T09:21:15.343 INFO:teuthology.orchestra.run.smithi164.stdout:(126/135): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-04-25T09:21:15.344 INFO:teuthology.orchestra.run.smithi043.stdout:(119/135): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-25T09:21:15.368 INFO:teuthology.orchestra.run.smithi164.stdout:(127/135): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-04-25T09:21:15.393 INFO:teuthology.orchestra.run.smithi164.stdout:(128/135): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-25T09:21:15.410 INFO:teuthology.orchestra.run.smithi164.stdout:(129/135): python3-zc-lockfile-2.0-10.el9.noarc 478 kB/s | 20 kB 00:00 2024-04-25T09:21:15.428 INFO:teuthology.orchestra.run.smithi043.stdout:(120/135): python3-repoze-lru-0.7-16.el9.noarch 369 kB/s | 31 kB 00:00 2024-04-25T09:21:15.435 INFO:teuthology.orchestra.run.smithi164.stdout:(130/135): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-04-25T09:21:15.486 INFO:teuthology.orchestra.run.smithi164.stdout:(131/135): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.4 MB/s | 122 kB 00:00 2024-04-25T09:21:15.586 INFO:teuthology.orchestra.run.smithi164.stdout:(132/135): python3-werkzeug-2.0.3-3.el9.1.noarc 1.7 MB/s | 427 kB 00:00 2024-04-25T09:21:15.619 INFO:teuthology.orchestra.run.smithi043.stdout:(121/135): python3-routes-2.5.1-5.el9.noarch.rp 985 kB/s | 188 kB 00:00 2024-04-25T09:21:15.703 INFO:teuthology.orchestra.run.smithi043.stdout:(122/135): python3-rsa-4.9-2.el9.noarch.rpm 710 kB/s | 59 kB 00:00 2024-04-25T09:21:15.770 INFO:teuthology.orchestra.run.smithi043.stdout:(123/135): python3-tempora-5.0.0-2.el9.noarch.r 537 kB/s | 36 kB 00:00 2024-04-25T09:21:15.786 INFO:teuthology.orchestra.run.smithi164.stdout:(133/135): thrift-0.15.0-2.el9.x86_64.rpm 4.2 MB/s | 1.6 MB 00:00 2024-04-25T09:21:15.837 INFO:teuthology.orchestra.run.smithi043.stdout:(124/135): python3-typing-extensions-4.4.0-2.el 763 kB/s | 51 kB 00:00 2024-04-25T09:21:15.912 INFO:teuthology.orchestra.run.smithi043.stdout:(125/135): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-25T09:21:16.013 INFO:teuthology.orchestra.run.smithi043.stdout:(126/135): python3-websocket-client-1.2.3-2.el9 896 kB/s | 90 kB 00:00 2024-04-25T09:21:16.130 INFO:teuthology.orchestra.run.smithi043.stdout:(127/135): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-25T09:21:16.196 INFO:teuthology.orchestra.run.smithi043.stdout:(128/135): python3-zc-lockfile-2.0-10.el9.noarc 301 kB/s | 20 kB 00:00 2024-04-25T09:21:16.272 INFO:teuthology.orchestra.run.smithi043.stdout:(129/135): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-25T09:21:16.363 INFO:teuthology.orchestra.run.smithi043.stdout:(130/135): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-25T09:21:16.447 INFO:teuthology.orchestra.run.smithi043.stdout:(131/135): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.4 MB/s | 122 kB 00:00 2024-04-25T09:21:16.686 INFO:teuthology.orchestra.run.smithi164.stdout:(134/135): librbd1-19.0.0-3194.ga126334d.el9.x8 3.3 MB/s | 3.7 MB 00:01 2024-04-25T09:21:17.297 INFO:teuthology.orchestra.run.smithi043.stdout:(132/135): librados2-19.0.0-3194.ga126334d.el9. 4.0 MB/s | 3.4 MB 00:00 2024-04-25T09:21:17.445 INFO:teuthology.orchestra.run.smithi164.stdout:(135/135): librados2-19.0.0-3194.ga126334d.el9. 1.7 MB/s | 3.4 MB 00:01 2024-04-25T09:21:17.448 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-04-25T09:21:17.449 INFO:teuthology.orchestra.run.smithi164.stdout:Total 12 MB/s | 292 MB 00:23 2024-04-25T09:21:17.873 INFO:teuthology.orchestra.run.smithi043.stdout:(133/135): librbd1-19.0.0-3194.ga126334d.el9.x8 6.4 MB/s | 3.7 MB 00:00 2024-04-25T09:21:18.296 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-04-25T09:21:18.403 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-04-25T09:21:18.403 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-04-25T09:21:20.491 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-04-25T09:21:20.491 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-04-25T09:21:20.731 INFO:teuthology.orchestra.run.smithi043.stdout:(134/135): ceph-osd-19.0.0-3194.ga126334d.el9.x 3.3 MB/s | 71 MB 00:21 2024-04-25T09:21:25.997 INFO:teuthology.orchestra.run.smithi043.stdout:(135/135): ceph-test-19.0.0-3194.ga126334d.el9. 3.3 MB/s | 84 MB 00:25 2024-04-25T09:21:26.001 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T09:21:26.002 INFO:teuthology.orchestra.run.smithi043.stdout:Total 9.0 MB/s | 292 MB 00:32 2024-04-25T09:21:26.235 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-04-25T09:21:26.305 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/137 2024-04-25T09:21:26.340 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/137 2024-04-25T09:21:26.391 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : liboath-2.6.7-2.el9.x86_64 3/137 2024-04-25T09:21:26.441 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/137 2024-04-25T09:21:26.898 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T09:21:26.900 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 5/137 2024-04-25T09:21:26.980 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librados2-2:19.0.0-3194.ga126334d.el9.x86_64 6/137 2024-04-25T09:21:27.015 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T09:21:27.015 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T09:21:27.106 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:19.0.0-3194.ga126334d.el9.x86_64 6/137 2024-04-25T09:21:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs2-2:19.0.0-3194.ga126334d.el9.x86_64 7/137 2024-04-25T09:21:27.232 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs2-2:19.0.0-3194.ga126334d.el9.x86_64 7/137 2024-04-25T09:21:27.275 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rados-2:19.0.0-3194.ga126334d.el9.x86_64 8/137 2024-04-25T09:21:27.304 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/137 2024-04-25T09:21:27.332 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 10/137 2024-04-25T09:21:27.684 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 11/137 2024-04-25T09:21:27.745 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librbd1-2:19.0.0-3194.ga126334d.el9.x86_64 12/137 2024-04-25T09:21:27.820 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:19.0.0-3194.ga126334d.el9.x86_64 12/137 2024-04-25T09:21:27.845 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephsqlite-2:19.0.0-3194.ga126334d.el9.x86_64 13/137 2024-04-25T09:21:27.944 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3194.ga126334d.el9.x86_64 13/137 2024-04-25T09:21:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libradosstriper1-2:19.0.0-3194.ga126334d.el9.x86 14/137 2024-04-25T09:21:28.034 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3194.ga126334d.el9.x86 14/137 2024-04-25T09:21:28.055 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/137 2024-04-25T09:21:28.075 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/137 2024-04-25T09:21:28.272 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/137 2024-04-25T09:21:28.312 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/137 2024-04-25T09:21:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 17/137 2024-04-25T09:21:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/137 2024-04-25T09:21:28.552 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/137 2024-04-25T09:21:28.595 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/137 2024-04-25T09:21:28.625 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 21/137 2024-04-25T09:21:28.666 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 22/137 2024-04-25T09:21:28.740 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/137 2024-04-25T09:21:28.792 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/137 2024-04-25T09:21:28.832 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/137 2024-04-25T09:21:28.866 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/137 2024-04-25T09:21:28.902 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/137 2024-04-25T09:21:28.932 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 28/137 2024-04-25T09:21:28.957 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 29/137 2024-04-25T09:21:29.004 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 29/137 2024-04-25T09:21:29.083 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/137 2024-04-25T09:21:29.134 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/137 2024-04-25T09:21:29.169 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/137 2024-04-25T09:21:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 33/137 2024-04-25T09:21:29.214 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T09:21:29.214 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T09:21:29.242 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 34/137 2024-04-25T09:21:29.319 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 35/137 2024-04-25T09:21:29.361 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-common-2:19.0.0-3194.ga126334d.el9. 36/137 2024-04-25T09:21:29.412 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-argparse-2:19.0.0-3194.ga126334d.el 37/137 2024-04-25T09:21:29.441 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cephfs-2:19.0.0-3194.ga126334d.el9.x86_6 38/137 2024-04-25T09:21:29.512 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 39/137 2024-04-25T09:21:29.588 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : luarocks-3.9.2-1.el9.noarch 40/137 2024-04-25T09:21:29.725 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/137 2024-04-25T09:21:29.774 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/137 2024-04-25T09:21:30.301 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/137 2024-04-25T09:21:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/137 2024-04-25T09:21:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 45/137 2024-04-25T09:21:30.475 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 46/137 2024-04-25T09:21:30.510 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 47/137 2024-04-25T09:21:30.542 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 48/137 2024-04-25T09:21:30.592 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 49/137 2024-04-25T09:21:30.643 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rbd-2:19.0.0-3194.ga126334d.el9.x86_64 50/137 2024-04-25T09:21:30.689 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados-devel-2:19.0.0-3194.ga126334d.el9.x86_6 51/137 2024-04-25T09:21:30.724 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 52/137 2024-04-25T09:21:30.754 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/137 2024-04-25T09:21:30.809 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 54/137 2024-04-25T09:21:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : thrift-0.15.0-2.el9.x86_64 55/137 2024-04-25T09:21:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 56/137 2024-04-25T09:21:30.971 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 57/137 2024-04-25T09:21:31.008 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 58/137 2024-04-25T09:21:31.045 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 59/137 2024-04-25T09:21:31.093 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 60/137 2024-04-25T09:21:31.137 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 61/137 2024-04-25T09:21:31.179 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 62/137 2024-04-25T09:21:31.320 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 63/137 2024-04-25T09:21:31.356 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 64/137 2024-04-25T09:21:31.397 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 65/137 2024-04-25T09:21:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 66/137 2024-04-25T09:21:31.530 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 67/137 2024-04-25T09:21:31.571 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 68/137 2024-04-25T09:21:31.596 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 69/137 2024-04-25T09:21:31.621 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 70/137 2024-04-25T09:21:31.697 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 71/137 2024-04-25T09:21:31.825 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 72/137 2024-04-25T09:21:31.863 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 73/137 2024-04-25T09:21:31.908 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 74/137 2024-04-25T09:21:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 75/137 2024-04-25T09:21:32.128 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-py-1.10.0-6.el9.noarch 76/137 2024-04-25T09:21:32.170 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/137 2024-04-25T09:21:32.201 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/137 2024-04-25T09:21:32.240 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/137 2024-04-25T09:21:32.340 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pluggy-0.13.1-7.el9.noarch 80/137 2024-04-25T09:21:32.394 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/137 2024-04-25T09:21:33.128 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/137 2024-04-25T09:21:33.174 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/137 2024-04-25T09:21:33.208 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/137 2024-04-25T09:21:33.242 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/137 2024-04-25T09:21:33.341 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-iniconfig-1.1.1-7.el9.noarch 86/137 2024-04-25T09:21:33.400 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 87/137 2024-04-25T09:21:33.425 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 88/137 2024-04-25T09:21:33.610 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/137 2024-04-25T09:21:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/137 2024-04-25T09:21:33.729 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/137 2024-04-25T09:21:34.379 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/137 2024-04-25T09:21:34.491 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/137 2024-04-25T09:21:35.227 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T09:21:35.283 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/137 2024-04-25T09:21:35.320 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/137 2024-04-25T09:21:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : liboath-2.6.7-2.el9.x86_64 3/137 2024-04-25T09:21:35.428 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/137 2024-04-25T09:21:35.748 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/137 2024-04-25T09:21:35.819 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/137 2024-04-25T09:21:35.863 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 96/137 2024-04-25T09:21:35.906 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 5/137 2024-04-25T09:21:35.976 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librados2-2:19.0.0-3194.ga126334d.el9.x86_64 6/137 2024-04-25T09:21:36.086 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:19.0.0-3194.ga126334d.el9.x86_64 6/137 2024-04-25T09:21:36.119 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs2-2:19.0.0-3194.ga126334d.el9.x86_64 7/137 2024-04-25T09:21:36.211 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs2-2:19.0.0-3194.ga126334d.el9.x86_64 7/137 2024-04-25T09:21:36.254 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rados-2:19.0.0-3194.ga126334d.el9.x86_64 8/137 2024-04-25T09:21:36.282 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/137 2024-04-25T09:21:36.318 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 10/137 2024-04-25T09:21:36.432 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 97/137 2024-04-25T09:21:36.475 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librgw2-2:19.0.0-3194.ga126334d.el9.x86_64 98/137 2024-04-25T09:21:36.546 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librgw2-2:19.0.0-3194.ga126334d.el9.x86_64 98/137 2024-04-25T09:21:36.576 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rgw-2:19.0.0-3194.ga126334d.el9.x86_64 99/137 2024-04-25T09:21:36.687 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 11/137 2024-04-25T09:21:36.739 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librbd1-2:19.0.0-3194.ga126334d.el9.x86_64 12/137 2024-04-25T09:21:36.799 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:19.0.0-3194.ga126334d.el9.x86_64 12/137 2024-04-25T09:21:36.823 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephsqlite-2:19.0.0-3194.ga126334d.el9.x86_64 13/137 2024-04-25T09:21:36.943 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3194.ga126334d.el9.x86_64 13/137 2024-04-25T09:21:36.974 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libradosstriper1-2:19.0.0-3194.ga126334d.el9.x86 14/137 2024-04-25T09:21:37.038 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3194.ga126334d.el9.x86 14/137 2024-04-25T09:21:37.067 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/137 2024-04-25T09:21:37.088 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/137 2024-04-25T09:21:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/137 2024-04-25T09:21:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/137 2024-04-25T09:21:37.359 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 17/137 2024-04-25T09:21:37.471 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/137 2024-04-25T09:21:37.568 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/137 2024-04-25T09:21:37.616 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/137 2024-04-25T09:21:37.645 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 21/137 2024-04-25T09:21:37.695 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 22/137 2024-04-25T09:21:37.771 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/137 2024-04-25T09:21:37.821 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/137 2024-04-25T09:21:37.868 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/137 2024-04-25T09:21:37.902 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/137 2024-04-25T09:21:37.930 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/137 2024-04-25T09:21:37.961 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 28/137 2024-04-25T09:21:37.977 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 29/137 2024-04-25T09:21:38.024 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 29/137 2024-04-25T09:21:38.097 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/137 2024-04-25T09:21:38.164 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/137 2024-04-25T09:21:38.198 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/137 2024-04-25T09:21:38.227 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 33/137 2024-04-25T09:21:38.271 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 34/137 2024-04-25T09:21:38.351 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 35/137 2024-04-25T09:21:38.382 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-common-2:19.0.0-3194.ga126334d.el9. 36/137 2024-04-25T09:21:38.426 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-argparse-2:19.0.0-3194.ga126334d.el 37/137 2024-04-25T09:21:38.454 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cephfs-2:19.0.0-3194.ga126334d.el9.x86_6 38/137 2024-04-25T09:21:38.529 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 39/137 2024-04-25T09:21:38.614 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : luarocks-3.9.2-1.el9.noarch 40/137 2024-04-25T09:21:38.762 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/137 2024-04-25T09:21:38.814 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/137 2024-04-25T09:21:38.991 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-3194.ga126334d.el9.x86_64 100/137 2024-04-25T09:21:39.031 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-common-2:19.0.0-3194.ga126334d.el9.x86_64 100/137 2024-04-25T09:21:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/137 2024-04-25T09:21:39.470 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/137 2024-04-25T09:21:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 45/137 2024-04-25T09:21:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 46/137 2024-04-25T09:21:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 47/137 2024-04-25T09:21:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 48/137 2024-04-25T09:21:39.674 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 49/137 2024-04-25T09:21:39.705 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-3194.ga126334d.el9.x86_64 100/137 2024-04-25T09:21:39.715 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rbd-2:19.0.0-3194.ga126334d.el9.x86_64 50/137 2024-04-25T09:21:39.743 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-base-2:19.0.0-3194.ga126334d.el9.x86_64 101/137 2024-04-25T09:21:39.761 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librados-devel-2:19.0.0-3194.ga126334d.el9.x86_6 51/137 2024-04-25T09:21:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 52/137 2024-04-25T09:21:39.818 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/137 2024-04-25T09:21:39.832 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-base-2:19.0.0-3194.ga126334d.el9.x86_64 101/137 2024-04-25T09:21:39.832 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T09:21:39.832 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T09:21:39.832 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:21:39.861 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-selinux-2:19.0.0-3194.ga126334d.el9.x86_64 102/137 2024-04-25T09:21:39.874 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 54/137 2024-04-25T09:21:39.906 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : thrift-0.15.0-2.el9.x86_64 55/137 2024-04-25T09:21:39.963 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 56/137 2024-04-25T09:21:40.037 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 57/137 2024-04-25T09:21:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 58/137 2024-04-25T09:21:40.110 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 59/137 2024-04-25T09:21:40.158 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 60/137 2024-04-25T09:21:40.202 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 61/137 2024-04-25T09:21:40.245 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 62/137 2024-04-25T09:21:40.393 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 63/137 2024-04-25T09:21:40.438 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 64/137 2024-04-25T09:21:40.480 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 65/137 2024-04-25T09:21:40.580 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 66/137 2024-04-25T09:21:40.612 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 67/137 2024-04-25T09:21:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 68/137 2024-04-25T09:21:40.686 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 69/137 2024-04-25T09:21:40.711 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 70/137 2024-04-25T09:21:40.782 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 71/137 2024-04-25T09:21:40.930 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 72/137 2024-04-25T09:21:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 73/137 2024-04-25T09:21:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 74/137 2024-04-25T09:21:41.149 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 75/137 2024-04-25T09:21:41.240 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-py-1.10.0-6.el9.noarch 76/137 2024-04-25T09:21:41.296 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/137 2024-04-25T09:21:41.325 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/137 2024-04-25T09:21:41.365 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/137 2024-04-25T09:21:41.480 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pluggy-0.13.1-7.el9.noarch 80/137 2024-04-25T09:21:41.554 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/137 2024-04-25T09:21:42.362 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/137 2024-04-25T09:21:42.457 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/137 2024-04-25T09:21:42.483 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/137 2024-04-25T09:21:42.516 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/137 2024-04-25T09:21:42.623 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-iniconfig-1.1.1-7.el9.noarch 86/137 2024-04-25T09:21:42.687 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 87/137 2024-04-25T09:21:42.718 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 88/137 2024-04-25T09:21:42.905 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/137 2024-04-25T09:21:42.932 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/137 2024-04-25T09:21:43.013 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/137 2024-04-25T09:21:43.722 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/137 2024-04-25T09:21:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/137 2024-04-25T09:21:45.221 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/137 2024-04-25T09:21:45.304 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/137 2024-04-25T09:21:45.348 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 96/137 2024-04-25T09:21:45.948 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 97/137 2024-04-25T09:21:45.976 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librgw2-2:19.0.0-3194.ga126334d.el9.x86_64 98/137 2024-04-25T09:21:46.048 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librgw2-2:19.0.0-3194.ga126334d.el9.x86_64 98/137 2024-04-25T09:21:46.078 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rgw-2:19.0.0-3194.ga126334d.el9.x86_64 99/137 2024-04-25T09:21:48.601 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:19.0.0-3194.ga126334d.el9.x86_64 100/137 2024-04-25T09:21:48.641 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-common-2:19.0.0-3194.ga126334d.el9.x86_64 100/137 2024-04-25T09:21:49.344 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:19.0.0-3194.ga126334d.el9.x86_64 100/137 2024-04-25T09:21:49.386 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-base-2:19.0.0-3194.ga126334d.el9.x86_64 101/137 2024-04-25T09:21:49.461 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-base-2:19.0.0-3194.ga126334d.el9.x86_64 101/137 2024-04-25T09:21:49.461 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T09:21:49.461 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T09:21:49.461 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:21:49.495 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-selinux-2:19.0.0-3194.ga126334d.el9.x86_64 102/137 2024-04-25T09:21:51.657 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3194.ga126334d.el9.x86_64 102/137 2024-04-25T09:21:51.657 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /sys 2024-04-25T09:21:51.657 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /proc 2024-04-25T09:21:51.657 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /mnt 2024-04-25T09:21:51.657 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /var/tmp 2024-04-25T09:21:51.657 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /home 2024-04-25T09:21:51.658 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /root 2024-04-25T09:21:51.658 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /tmp 2024-04-25T09:21:51.658 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:21:51.865 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-volume-2:19.0.0-3194.ga126334d.el9.noarch 103/137 2024-04-25T09:21:51.880 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-volume-2:19.0.0-3194.ga126334d.el9.noarch 103/137 2024-04-25T09:21:51.880 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:21:51.880 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T09:21:51.881 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:01.705 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3194.ga126334d.el9.x86_64 102/137 2024-04-25T09:22:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /sys 2024-04-25T09:22:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /proc 2024-04-25T09:22:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /mnt 2024-04-25T09:22:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /var/tmp 2024-04-25T09:22:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /home 2024-04-25T09:22:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /root 2024-04-25T09:22:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /tmp 2024-04-25T09:22:01.706 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:01.883 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-volume-2:19.0.0-3194.ga126334d.el9.noarch 103/137 2024-04-25T09:22:01.899 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-volume-2:19.0.0-3194.ga126334d.el9.noarch 103/137 2024-04-25T09:22:01.899 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:01.900 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T09:22:01.900 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:02.815 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-osd-2:19.0.0-3194.ga126334d.el9.x86_64 104/137 2024-04-25T09:22:02.840 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-osd-2:19.0.0-3194.ga126334d.el9.x86_64 104/137 2024-04-25T09:22:02.840 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:02.840 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T09:22:02.840 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T09:22:02.840 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T09:22:02.840 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:03.074 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mds-2:19.0.0-3194.ga126334d.el9.x86_64 105/137 2024-04-25T09:22:03.096 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mds-2:19.0.0-3194.ga126334d.el9.x86_64 105/137 2024-04-25T09:22:03.097 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:03.097 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T09:22:03.097 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T09:22:03.097 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T09:22:03.097 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:03.633 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mon-2:19.0.0-3194.ga126334d.el9.x86_64 106/137 2024-04-25T09:22:03.655 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mon-2:19.0.0-3194.ga126334d.el9.x86_64 106/137 2024-04-25T09:22:03.655 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:03.655 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T09:22:03.655 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T09:22:03.656 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T09:22:03.656 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:03.720 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ply-3.11-14.el9.noarch 107/137 2024-04-25T09:22:03.774 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 108/137 2024-04-25T09:22:03.959 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 109/137 2024-04-25T09:22:04.008 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 110/137 2024-04-25T09:22:04.085 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 111/137 2024-04-25T09:22:04.182 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 112/137 2024-04-25T09:22:04.321 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 113/137 2024-04-25T09:22:04.358 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 114/137 2024-04-25T09:22:04.390 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 115/137 2024-04-25T09:22:04.417 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-2.9.9-15.el9.x86_64 116/137 2024-04-25T09:22:04.870 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:19.0.0-3194.ga126334d.el9.noarch 117/137 2024-04-25T09:22:04.899 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephadm-2:19.0.0-3194.ga126334d.el9.noarch 117/137 2024-04-25T09:22:04.931 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3194.ga126334d.e 118/137 2024-04-25T09:22:04.992 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3194.ga126334d. 119/137 2024-04-25T09:22:05.022 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3194.ga126334d.el9.noa 120/137 2024-04-25T09:22:06.066 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3194.ga126334d.el9.noa 120/137 2024-04-25T09:22:06.107 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3194.ga126334d.el9.n 121/137 2024-04-25T09:22:07.078 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3194.ga126334d.el9.n 121/137 2024-04-25T09:22:07.108 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3194.ga12 122/137 2024-04-25T09:22:07.208 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3194.ga12 122/137 2024-04-25T09:22:07.399 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3194.ga126334d.el 123/137 2024-04-25T09:22:07.426 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-2:19.0.0-3194.ga126334d.el9.x86_64 124/137 2024-04-25T09:22:07.449 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3194.ga126334d.el9.x86_64 124/137 2024-04-25T09:22:07.449 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:07.449 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T09:22:07.449 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T09:22:07.449 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T09:22:07.449 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:07.502 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-rook-2:19.0.0-3194.ga126334d.el9.noarch 125/137 2024-04-25T09:22:07.513 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3194.ga126334d.el9.noarch 125/137 2024-04-25T09:22:07.619 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-2:19.0.0-3194.ga126334d.el9.x86_64 126/137 2024-04-25T09:22:08.634 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-fuse-2:19.0.0-3194.ga126334d.el9.x86_64 127/137 2024-04-25T09:22:08.670 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-radosgw-2:19.0.0-3194.ga126334d.el9.x86_64 128/137 2024-04-25T09:22:08.693 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3194.ga126334d.el9.x86_64 128/137 2024-04-25T09:22:08.693 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:08.693 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T09:22:08.693 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T09:22:08.693 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T09:22:08.693 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:08.738 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3194.ga1263 129/137 2024-04-25T09:22:08.759 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3194.ga1263 129/137 2024-04-25T09:22:08.759 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:08.759 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T09:22:08.759 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-mirror-2:19.0.0-3194.ga126334d.el9.x86_64 130/137 2024-04-25T09:22:09.219 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3194.ga126334d.el9.x86_64 130/137 2024-04-25T09:22:09.219 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:09.219 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T09:22:09.220 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T09:22:09.220 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T09:22:09.220 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:13.419 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-osd-2:19.0.0-3194.ga126334d.el9.x86_64 104/137 2024-04-25T09:22:13.447 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-osd-2:19.0.0-3194.ga126334d.el9.x86_64 104/137 2024-04-25T09:22:13.447 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:13.448 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T09:22:13.448 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T09:22:13.448 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T09:22:13.448 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mds-2:19.0.0-3194.ga126334d.el9.x86_64 105/137 2024-04-25T09:22:13.719 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mds-2:19.0.0-3194.ga126334d.el9.x86_64 105/137 2024-04-25T09:22:13.719 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:13.719 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T09:22:13.719 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T09:22:13.719 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T09:22:13.719 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:14.279 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mon-2:19.0.0-3194.ga126334d.el9.x86_64 106/137 2024-04-25T09:22:14.302 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mon-2:19.0.0-3194.ga126334d.el9.x86_64 106/137 2024-04-25T09:22:14.302 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:14.303 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T09:22:14.303 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T09:22:14.303 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T09:22:14.303 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:14.375 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ply-3.11-14.el9.noarch 107/137 2024-04-25T09:22:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 108/137 2024-04-25T09:22:14.643 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 109/137 2024-04-25T09:22:14.687 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 110/137 2024-04-25T09:22:14.768 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 111/137 2024-04-25T09:22:14.858 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 112/137 2024-04-25T09:22:15.018 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 113/137 2024-04-25T09:22:15.062 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 114/137 2024-04-25T09:22:15.094 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 115/137 2024-04-25T09:22:15.147 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : fuse-2.9.9-15.el9.x86_64 116/137 2024-04-25T09:22:15.575 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:19.0.0-3194.ga126334d.el9.noarch 117/137 2024-04-25T09:22:15.604 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephadm-2:19.0.0-3194.ga126334d.el9.noarch 117/137 2024-04-25T09:22:15.636 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3194.ga126334d.e 118/137 2024-04-25T09:22:15.699 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3194.ga126334d. 119/137 2024-04-25T09:22:15.727 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3194.ga126334d.el9.noa 120/137 2024-04-25T09:22:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3194.ga126334d.el9.noa 120/137 2024-04-25T09:22:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3194.ga126334d.el9.n 121/137 2024-04-25T09:22:17.882 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3194.ga126334d.el9.n 121/137 2024-04-25T09:22:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3194.ga12 122/137 2024-04-25T09:22:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3194.ga12 122/137 2024-04-25T09:22:18.212 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3194.ga126334d.el 123/137 2024-04-25T09:22:18.256 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-2:19.0.0-3194.ga126334d.el9.x86_64 124/137 2024-04-25T09:22:18.280 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3194.ga126334d.el9.x86_64 124/137 2024-04-25T09:22:18.280 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:18.281 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T09:22:18.281 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T09:22:18.281 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T09:22:18.281 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-rook-2:19.0.0-3194.ga126334d.el9.noarch 125/137 2024-04-25T09:22:18.343 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3194.ga126334d.el9.noarch 125/137 2024-04-25T09:22:18.454 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-2:19.0.0-3194.ga126334d.el9.x86_64 126/137 2024-04-25T09:22:19.544 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-fuse-2:19.0.0-3194.ga126334d.el9.x86_64 127/137 2024-04-25T09:22:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-radosgw-2:19.0.0-3194.ga126334d.el9.x86_64 128/137 2024-04-25T09:22:19.599 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3194.ga126334d.el9.x86_64 128/137 2024-04-25T09:22:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T09:22:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T09:22:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T09:22:19.600 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:19.643 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3194.ga1263 129/137 2024-04-25T09:22:19.666 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3194.ga1263 129/137 2024-04-25T09:22:19.666 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:19.666 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T09:22:19.666 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:20.127 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-mirror-2:19.0.0-3194.ga126334d.el9.x86_64 130/137 2024-04-25T09:22:20.150 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3194.ga126334d.el9.x86_64 130/137 2024-04-25T09:22:20.150 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T09:22:20.150 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T09:22:20.151 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T09:22:20.151 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T09:22:20.151 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:20.585 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-test-2:19.0.0-3194.ga126334d.el9.x86_64 131/137 2024-04-25T09:22:20.623 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pytest-6.2.2-7.el9.noarch 132/137 2024-04-25T09:22:20.660 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs-devel-2:19.0.0-3194.ga126334d.el9.x86_ 133/137 2024-04-25T09:22:20.702 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-fuse-2:19.0.0-3194.ga126334d.el9.x86_64 134/137 2024-04-25T09:22:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-nbd-2:19.0.0-3194.ga126334d.el9.x86_64 135/137 2024-04-25T09:22:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 136/137 2024-04-25T09:22:20.813 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 136/137 2024-04-25T09:22:20.813 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 137/137 2024-04-25T09:22:32.239 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-test-2:19.0.0-3194.ga126334d.el9.x86_64 131/137 2024-04-25T09:22:32.278 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pytest-6.2.2-7.el9.noarch 132/137 2024-04-25T09:22:32.316 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs-devel-2:19.0.0-3194.ga126334d.el9.x86_ 133/137 2024-04-25T09:22:32.358 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-fuse-2:19.0.0-3194.ga126334d.el9.x86_64 134/137 2024-04-25T09:22:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-nbd-2:19.0.0-3194.ga126334d.el9.x86_64 135/137 2024-04-25T09:22:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 136/137 2024-04-25T09:22:32.469 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 136/137 2024-04-25T09:22:32.469 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 137/137 2024-04-25T09:22:35.215 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 137/137 2024-04-25T09:22:35.215 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-2:19.0.0-3194.ga126334d.el9.x86_64 1/137 2024-04-25T09:22:35.215 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-base-2:19.0.0-3194.ga126334d.el9.x86_64 2/137 2024-04-25T09:22:35.215 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-common-2:19.0.0-3194.ga126334d.el9.x86_64 3/137 2024-04-25T09:22:35.215 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-fuse-2:19.0.0-3194.ga126334d.el9.x86_64 4/137 2024-04-25T09:22:35.215 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3194.ga1263 5/137 2024-04-25T09:22:35.215 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mds-2:19.0.0-3194.ga126334d.el9.x86_64 6/137 2024-04-25T09:22:35.215 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-2:19.0.0-3194.ga126334d.el9.x86_64 7/137 2024-04-25T09:22:35.216 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mon-2:19.0.0-3194.ga126334d.el9.x86_64 8/137 2024-04-25T09:22:35.216 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-osd-2:19.0.0-3194.ga126334d.el9.x86_64 9/137 2024-04-25T09:22:35.216 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-radosgw-2:19.0.0-3194.ga126334d.el9.x86_64 10/137 2024-04-25T09:22:35.216 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-selinux-2:19.0.0-3194.ga126334d.el9.x86_64 11/137 2024-04-25T09:22:35.216 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-test-2:19.0.0-3194.ga126334d.el9.x86_64 12/137 2024-04-25T09:22:35.216 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs-devel-2:19.0.0-3194.ga126334d.el9.x86_ 13/137 2024-04-25T09:22:35.216 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs2-2:19.0.0-3194.ga126334d.el9.x86_64 14/137 2024-04-25T09:22:35.216 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephsqlite-2:19.0.0-3194.ga126334d.el9.x86_64 15/137 2024-04-25T09:22:35.216 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados-devel-2:19.0.0-3194.ga126334d.el9.x86_6 16/137 2024-04-25T09:22:35.216 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libradosstriper1-2:19.0.0-3194.ga126334d.el9.x86 17/137 2024-04-25T09:22:35.216 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librgw2-2:19.0.0-3194.ga126334d.el9.x86_64 18/137 2024-04-25T09:22:35.216 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3194.ga126334d.el 19/137 2024-04-25T09:22:35.216 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-common-2:19.0.0-3194.ga126334d.el9. 20/137 2024-04-25T09:22:35.217 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cephfs-2:19.0.0-3194.ga126334d.el9.x86_6 21/137 2024-04-25T09:22:35.217 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rados-2:19.0.0-3194.ga126334d.el9.x86_64 22/137 2024-04-25T09:22:35.217 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rbd-2:19.0.0-3194.ga126334d.el9.x86_64 23/137 2024-04-25T09:22:35.217 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rgw-2:19.0.0-3194.ga126334d.el9.x86_64 24/137 2024-04-25T09:22:35.217 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-fuse-2:19.0.0-3194.ga126334d.el9.x86_64 25/137 2024-04-25T09:22:35.217 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-mirror-2:19.0.0-3194.ga126334d.el9.x86_64 26/137 2024-04-25T09:22:35.227 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-nbd-2:19.0.0-3194.ga126334d.el9.x86_64 27/137 2024-04-25T09:22:35.227 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3194.ga126334d. 28/137 2024-04-25T09:22:35.227 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3194.ga126334d.el9.noa 29/137 2024-04-25T09:22:35.227 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3194.ga126334d.el9.n 30/137 2024-04-25T09:22:35.227 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3194.ga12 31/137 2024-04-25T09:22:35.227 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3194.ga126334d.el 32/137 2024-04-25T09:22:35.227 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3194.ga126334d.el9.noarch 33/137 2024-04-25T09:22:35.227 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3194.ga126334d.e 34/137 2024-04-25T09:22:35.227 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-volume-2:19.0.0-3194.ga126334d.el9.noarch 35/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephadm-2:19.0.0-3194.ga126334d.el9.noarch 36/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 37/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 38/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 39/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 40/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 41/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 42/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ply-3.11-14.el9.noarch 43/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 44/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 45/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 46/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 47/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 48/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 49/137 2024-04-25T09:22:35.228 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 50/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 51/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 52/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 53/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 54/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 55/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 56/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-5.4.4-4.el9.x86_64 57/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 58/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 59/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 60/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 61/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 62/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 63/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-iniconfig-1.1.1-7.el9.noarch 64/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 65/137 2024-04-25T09:22:35.229 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 66/137 2024-04-25T09:22:35.230 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 67/137 2024-04-25T09:22:35.230 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 68/137 2024-04-25T09:22:35.230 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 69/137 2024-04-25T09:22:35.230 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 70/137 2024-04-25T09:22:35.230 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 71/137 2024-04-25T09:22:35.230 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 72/137 2024-04-25T09:22:35.230 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 73/137 2024-04-25T09:22:35.230 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pluggy-0.13.1-7.el9.noarch 74/137 2024-04-25T09:22:35.230 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 75/137 2024-04-25T09:22:35.230 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 76/137 2024-04-25T09:22:35.230 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-py-1.10.0-6.el9.noarch 77/137 2024-04-25T09:22:35.230 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/137 2024-04-25T09:22:35.230 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pytest-6.2.2-7.el9.noarch 80/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 81/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 83/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 84/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 85/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 86/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 87/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 89/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 90/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 91/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 92/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 93/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 94/137 2024-04-25T09:22:35.231 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 95/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 96/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 97/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 98/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 99/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 100/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 101/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 102/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 103/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 104/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 105/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 106/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 107/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 108/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 109/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 110/137 2024-04-25T09:22:35.232 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 111/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 112/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 113/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 114/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 115/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 116/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 117/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 118/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 119/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 120/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 122/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 123/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 124/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 125/137 2024-04-25T09:22:35.233 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 126/137 2024-04-25T09:22:35.234 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 127/137 2024-04-25T09:22:35.234 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 128/137 2024-04-25T09:22:35.234 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 129/137 2024-04-25T09:22:35.234 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 130/137 2024-04-25T09:22:35.234 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : re2-1:20211101-3.el9.x86_64 131/137 2024-04-25T09:22:35.234 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 132/137 2024-04-25T09:22:35.234 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 133/137 2024-04-25T09:22:35.234 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:19.0.0-3194.ga126334d.el9.x86_64 134/137 2024-04-25T09:22:35.234 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 135/137 2024-04-25T09:22:35.234 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:19.0.0-3194.ga126334d.el9.x86_64 136/137 2024-04-25T09:22:37.731 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 137/137 2024-04-25T09:22:37.731 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:37.731 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2024-04-25T09:22:37.731 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.731 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.731 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-04-25T09:22:37.731 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T09:22:37.731 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T09:22:37.731 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.731 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.731 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.732 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.733 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-25T09:22:37.734 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T09:22:37.735 INFO:teuthology.orchestra.run.smithi164.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T09:22:37.735 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T09:22:37.735 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T09:22:37.735 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T09:22:37.735 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T09:22:37.735 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T09:22:37.735 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T09:22:37.735 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T09:22:37.735 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T09:22:37.736 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-iniconfig-1.1.1-7.el9.noarch 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T09:22:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pluggy-0.13.1-7.el9.noarch 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T09:22:37.738 INFO:teuthology.orchestra.run.smithi164.stdout: python3-py-1.10.0-6.el9.noarch 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytest-6.2.2-7.el9.noarch 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T09:22:37.739 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.740 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:37.741 INFO:teuthology.orchestra.run.smithi164.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T09:22:37.741 INFO:teuthology.orchestra.run.smithi164.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T09:22:37.741 INFO:teuthology.orchestra.run.smithi164.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T09:22:37.741 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T09:22:37.741 INFO:teuthology.orchestra.run.smithi164.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-25T09:22:37.741 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:37.741 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-04-25T09:22:38.178 DEBUG:teuthology.parallel:result is None 2024-04-25T09:22:47.232 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 137/137 2024-04-25T09:22:47.233 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-2:19.0.0-3194.ga126334d.el9.x86_64 1/137 2024-04-25T09:22:47.233 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-base-2:19.0.0-3194.ga126334d.el9.x86_64 2/137 2024-04-25T09:22:47.233 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-common-2:19.0.0-3194.ga126334d.el9.x86_64 3/137 2024-04-25T09:22:47.233 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-fuse-2:19.0.0-3194.ga126334d.el9.x86_64 4/137 2024-04-25T09:22:47.233 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3194.ga1263 5/137 2024-04-25T09:22:47.233 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mds-2:19.0.0-3194.ga126334d.el9.x86_64 6/137 2024-04-25T09:22:47.233 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-2:19.0.0-3194.ga126334d.el9.x86_64 7/137 2024-04-25T09:22:47.233 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mon-2:19.0.0-3194.ga126334d.el9.x86_64 8/137 2024-04-25T09:22:47.233 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-osd-2:19.0.0-3194.ga126334d.el9.x86_64 9/137 2024-04-25T09:22:47.234 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-radosgw-2:19.0.0-3194.ga126334d.el9.x86_64 10/137 2024-04-25T09:22:47.234 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-selinux-2:19.0.0-3194.ga126334d.el9.x86_64 11/137 2024-04-25T09:22:47.234 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-test-2:19.0.0-3194.ga126334d.el9.x86_64 12/137 2024-04-25T09:22:47.234 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs-devel-2:19.0.0-3194.ga126334d.el9.x86_ 13/137 2024-04-25T09:22:47.234 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs2-2:19.0.0-3194.ga126334d.el9.x86_64 14/137 2024-04-25T09:22:47.234 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephsqlite-2:19.0.0-3194.ga126334d.el9.x86_64 15/137 2024-04-25T09:22:47.234 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados-devel-2:19.0.0-3194.ga126334d.el9.x86_6 16/137 2024-04-25T09:22:47.234 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libradosstriper1-2:19.0.0-3194.ga126334d.el9.x86 17/137 2024-04-25T09:22:47.234 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librgw2-2:19.0.0-3194.ga126334d.el9.x86_64 18/137 2024-04-25T09:22:47.234 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3194.ga126334d.el 19/137 2024-04-25T09:22:47.235 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-common-2:19.0.0-3194.ga126334d.el9. 20/137 2024-04-25T09:22:47.235 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cephfs-2:19.0.0-3194.ga126334d.el9.x86_6 21/137 2024-04-25T09:22:47.235 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rados-2:19.0.0-3194.ga126334d.el9.x86_64 22/137 2024-04-25T09:22:47.235 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rbd-2:19.0.0-3194.ga126334d.el9.x86_64 23/137 2024-04-25T09:22:47.235 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rgw-2:19.0.0-3194.ga126334d.el9.x86_64 24/137 2024-04-25T09:22:47.235 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-fuse-2:19.0.0-3194.ga126334d.el9.x86_64 25/137 2024-04-25T09:22:47.236 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-mirror-2:19.0.0-3194.ga126334d.el9.x86_64 26/137 2024-04-25T09:22:47.236 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-nbd-2:19.0.0-3194.ga126334d.el9.x86_64 27/137 2024-04-25T09:22:47.236 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3194.ga126334d. 28/137 2024-04-25T09:22:47.236 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3194.ga126334d.el9.noa 29/137 2024-04-25T09:22:47.236 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3194.ga126334d.el9.n 30/137 2024-04-25T09:22:47.236 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3194.ga12 31/137 2024-04-25T09:22:47.237 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3194.ga126334d.el 32/137 2024-04-25T09:22:47.237 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3194.ga126334d.el9.noarch 33/137 2024-04-25T09:22:47.237 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3194.ga126334d.e 34/137 2024-04-25T09:22:47.237 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-volume-2:19.0.0-3194.ga126334d.el9.noarch 35/137 2024-04-25T09:22:47.237 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephadm-2:19.0.0-3194.ga126334d.el9.noarch 36/137 2024-04-25T09:22:47.237 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 37/137 2024-04-25T09:22:47.238 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 38/137 2024-04-25T09:22:47.238 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 39/137 2024-04-25T09:22:47.238 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 40/137 2024-04-25T09:22:47.238 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 41/137 2024-04-25T09:22:47.238 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 42/137 2024-04-25T09:22:47.238 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ply-3.11-14.el9.noarch 43/137 2024-04-25T09:22:47.238 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 44/137 2024-04-25T09:22:47.238 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 45/137 2024-04-25T09:22:47.238 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 46/137 2024-04-25T09:22:47.238 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 47/137 2024-04-25T09:22:47.238 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 48/137 2024-04-25T09:22:47.239 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 49/137 2024-04-25T09:22:47.239 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 50/137 2024-04-25T09:22:47.239 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 51/137 2024-04-25T09:22:47.239 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 52/137 2024-04-25T09:22:47.239 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 53/137 2024-04-25T09:22:47.239 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 54/137 2024-04-25T09:22:47.240 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 55/137 2024-04-25T09:22:47.240 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 56/137 2024-04-25T09:22:47.240 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-5.4.4-4.el9.x86_64 57/137 2024-04-25T09:22:47.240 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 58/137 2024-04-25T09:22:47.240 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 59/137 2024-04-25T09:22:47.240 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 60/137 2024-04-25T09:22:47.240 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 61/137 2024-04-25T09:22:47.240 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 62/137 2024-04-25T09:22:47.240 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 63/137 2024-04-25T09:22:47.240 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-iniconfig-1.1.1-7.el9.noarch 64/137 2024-04-25T09:22:47.240 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 65/137 2024-04-25T09:22:47.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 66/137 2024-04-25T09:22:47.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 67/137 2024-04-25T09:22:47.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 68/137 2024-04-25T09:22:47.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 69/137 2024-04-25T09:22:47.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 70/137 2024-04-25T09:22:47.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 71/137 2024-04-25T09:22:47.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 72/137 2024-04-25T09:22:47.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 73/137 2024-04-25T09:22:47.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pluggy-0.13.1-7.el9.noarch 74/137 2024-04-25T09:22:47.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 75/137 2024-04-25T09:22:47.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 76/137 2024-04-25T09:22:47.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-py-1.10.0-6.el9.noarch 77/137 2024-04-25T09:22:47.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/137 2024-04-25T09:22:47.242 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/137 2024-04-25T09:22:47.242 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pytest-6.2.2-7.el9.noarch 80/137 2024-04-25T09:22:47.242 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 81/137 2024-04-25T09:22:47.242 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/137 2024-04-25T09:22:47.242 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 83/137 2024-04-25T09:22:47.242 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 84/137 2024-04-25T09:22:47.242 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 85/137 2024-04-25T09:22:47.242 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 86/137 2024-04-25T09:22:47.242 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 87/137 2024-04-25T09:22:47.242 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/137 2024-04-25T09:22:47.242 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 89/137 2024-04-25T09:22:47.246 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 90/137 2024-04-25T09:22:47.246 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 91/137 2024-04-25T09:22:47.246 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 92/137 2024-04-25T09:22:47.246 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 93/137 2024-04-25T09:22:47.246 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 94/137 2024-04-25T09:22:47.247 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 95/137 2024-04-25T09:22:47.247 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 96/137 2024-04-25T09:22:47.247 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 97/137 2024-04-25T09:22:47.247 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 98/137 2024-04-25T09:22:47.247 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 99/137 2024-04-25T09:22:47.247 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 100/137 2024-04-25T09:22:47.247 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 101/137 2024-04-25T09:22:47.247 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 102/137 2024-04-25T09:22:47.247 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 103/137 2024-04-25T09:22:47.247 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 104/137 2024-04-25T09:22:47.247 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 105/137 2024-04-25T09:22:47.247 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 106/137 2024-04-25T09:22:47.247 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 107/137 2024-04-25T09:22:47.248 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 108/137 2024-04-25T09:22:47.248 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 109/137 2024-04-25T09:22:47.248 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 110/137 2024-04-25T09:22:47.248 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 111/137 2024-04-25T09:22:47.248 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 112/137 2024-04-25T09:22:47.248 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 113/137 2024-04-25T09:22:47.248 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 114/137 2024-04-25T09:22:47.248 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 115/137 2024-04-25T09:22:47.248 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 116/137 2024-04-25T09:22:47.248 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 117/137 2024-04-25T09:22:47.249 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 118/137 2024-04-25T09:22:47.249 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 119/137 2024-04-25T09:22:47.249 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 120/137 2024-04-25T09:22:47.249 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/137 2024-04-25T09:22:47.249 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 122/137 2024-04-25T09:22:47.249 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 123/137 2024-04-25T09:22:47.250 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 124/137 2024-04-25T09:22:47.250 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 125/137 2024-04-25T09:22:47.250 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 126/137 2024-04-25T09:22:47.250 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 127/137 2024-04-25T09:22:47.250 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 128/137 2024-04-25T09:22:47.250 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 129/137 2024-04-25T09:22:47.250 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 130/137 2024-04-25T09:22:47.250 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : re2-1:20211101-3.el9.x86_64 131/137 2024-04-25T09:22:47.250 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 132/137 2024-04-25T09:22:47.250 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 133/137 2024-04-25T09:22:47.250 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:19.0.0-3194.ga126334d.el9.x86_64 134/137 2024-04-25T09:22:47.250 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 135/137 2024-04-25T09:22:47.251 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:19.0.0-3194.ga126334d.el9.x86_64 136/137 2024-04-25T09:22:49.473 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 137/137 2024-04-25T09:22:49.474 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:49.474 INFO:teuthology.orchestra.run.smithi043.stdout:Upgraded: 2024-04-25T09:22:49.474 INFO:teuthology.orchestra.run.smithi043.stdout: librados2-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.474 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.474 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T09:22:49.474 INFO:teuthology.orchestra.run.smithi043.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T09:22:49.474 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T09:22:49.474 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.474 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.474 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.474 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.474 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.475 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:49.476 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm-2:19.0.0-3194.ga126334d.el9.noarch 2024-04-25T09:22:49.476 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T09:22:49.476 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T09:22:49.476 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T09:22:49.476 INFO:teuthology.orchestra.run.smithi043.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T09:22:49.476 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T09:22:49.476 INFO:teuthology.orchestra.run.smithi043.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T09:22:49.476 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T09:22:49.476 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T09:22:49.476 INFO:teuthology.orchestra.run.smithi043.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-25T09:22:49.476 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.476 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-25T09:22:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T09:22:49.478 INFO:teuthology.orchestra.run.smithi043.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T09:22:49.478 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T09:22:49.478 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T09:22:49.478 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T09:22:49.478 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T09:22:49.478 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T09:22:49.478 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T09:22:49.478 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T09:22:49.478 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T09:22:49.478 INFO:teuthology.orchestra.run.smithi043.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T09:22:49.478 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T09:22:49.478 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T09:22:49.478 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-iniconfig-1.1.1-7.el9.noarch 2024-04-25T09:22:49.479 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T09:22:49.480 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T09:22:49.481 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T09:22:49.481 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T09:22:49.481 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T09:22:49.481 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pluggy-0.13.1-7.el9.noarch 2024-04-25T09:22:49.481 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T09:22:49.481 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T09:22:49.481 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T09:22:49.481 INFO:teuthology.orchestra.run.smithi043.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T09:22:49.481 INFO:teuthology.orchestra.run.smithi043.stdout: python3-py-1.10.0-6.el9.noarch 2024-04-25T09:22:49.481 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T09:22:49.481 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T09:22:49.481 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T09:22:49.481 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T09:22:49.482 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytest-6.2.2-7.el9.noarch 2024-04-25T09:22:49.482 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T09:22:49.482 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.482 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.482 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T09:22:49.482 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T09:22:49.482 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T09:22:49.482 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.482 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T09:22:49.482 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T09:22:49.482 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T09:22:49.482 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd-2:19.0.0-3194.ga126334d.el9.x86_64 2024-04-25T09:22:49.483 INFO:teuthology.orchestra.run.smithi043.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T09:22:49.484 INFO:teuthology.orchestra.run.smithi043.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T09:22:49.484 INFO:teuthology.orchestra.run.smithi043.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T09:22:49.484 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T09:22:49.484 INFO:teuthology.orchestra.run.smithi043.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-25T09:22:49.484 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:49.484 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T09:22:49.855 DEBUG:teuthology.parallel:result is None 2024-04-25T09:22:49.856 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=a126334d8733673d119ffdf40d6ae4b541a0a1cd 2024-04-25T09:22:50.052 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T09:22:50.076 INFO:teuthology.orchestra.run.smithi043.stdout:19.0.0-3194.ga126334d.el9 2024-04-25T09:22:50.076 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3194.ga126334d.el9 2024-04-25T09:22:50.076 INFO:teuthology.task.install:The correct ceph version 19.0.0-3194.ga126334d is installed. 2024-04-25T09:22:50.078 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=a126334d8733673d119ffdf40d6ae4b541a0a1cd 2024-04-25T09:22:50.235 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T09:22:50.261 INFO:teuthology.orchestra.run.smithi164.stdout:19.0.0-3194.ga126334d.el9 2024-04-25T09:22:50.261 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3194.ga126334d.el9 2024-04-25T09:22:50.261 INFO:teuthology.task.install:The correct ceph version 19.0.0-3194.ga126334d is installed. 2024-04-25T09:22:50.263 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T09:22:50.263 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:22:50.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T09:22:50.292 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:22:50.292 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T09:22:50.330 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T09:22:50.330 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:22:50.330 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T09:22:50.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T09:22:50.424 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:22:50.424 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T09:22:50.452 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T09:22:50.520 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T09:22:50.520 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:22:50.520 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T09:22:50.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T09:22:50.611 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:22:50.611 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T09:22:50.637 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T09:22:50.703 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T09:22:50.703 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:22:50.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T09:22:50.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T09:22:50.794 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:22:50.794 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T09:22:50.819 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T09:22:50.886 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T09:22:50.977 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-25T09:22:50.977 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 777 /var/log/ceph 2024-04-25T09:22:50.980 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /var/log/ceph 2024-04-25T09:22:51.005 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-25T09:22:51.005 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T09:22:51.045 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T09:22:51.070 INFO:tasks.ceph:Creating extra log directories... 2024-04-25T09:22:51.070 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T09:22:51.110 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T09:22:51.146 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-25T09:22:51.146 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-25T09:22:51.146 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi/7672977', 'branch': 'wip-matanb-crimson-testing-april-22', 'description': 'crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python}', 'email': None, 'first_in_suite': False, 'job_id': '7672977', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-testing-april-22'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'a126334d8733673d119ffdf40d6ae4b541a0a1cd'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'extra_system_packages': ['python3-pytest'], 'flavor': 'crimson', 'sha1': 'a126334d8733673d119ffdf40d6ae4b541a0a1cd'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '472cc2b524a88306ef4535053d26658c5fd6e9db'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 1211, 'sha1': 'a126334d8733673d119ffdf40d6ae4b541a0a1cd', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_472cc2b524a88306ef4535053d26658c5fd6e9db/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '472cc2b524a88306ef4535053d26658c5fd6e9db', 'targets': {'smithi043.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ/waeR5lfqj8M3qpqR8FrH+LBOPZPD4bWUQFEmy26+gU96AWDxesKNhELsdhnzJBJyJTGZG+p+ff4mUgcF9jVY=', 'smithi164.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMITCQg/P/TXatLUkWpLdWOo+PWhMEbA40XRXgSSf2pNhLX2C2SRiY5OGznGrRQ5ieeVutHA2JKHKLiIi31Llzg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'a126334d8733673d119ffdf40d6ae4b541a0a1cd', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ["rados/test_python.sh -m 'not (wait or tier or ec or bench or stats)'"]}, 'timeout': '1h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-25_08:07:11', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T09:22:51.147 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T09:22:51.175 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T09:22:51.191 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T09:22:51.229 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T09:22:51.255 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:22:51.255 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T09:22:51.307 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T09:22:51.307 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_1 2024-04-25T09:22:51.362 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T09:22:51.362 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T09:22:51.363 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-25T09:22:51.363 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T09:22:51.363 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T09:22:51.363 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-25 09:22:32.973972129 +0000 2024-04-25T09:22:51.363 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-25 09:20:13.408541678 +0000 2024-04-25T09:22:51.363 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-25 09:20:13.408541678 +0000 2024-04-25T09:22:51.363 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-04-25 09:20:13.408541678 +0000 2024-04-25T09:22:51.363 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T09:22:51.426 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-25T09:22:51.426 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-25T09:22:51.426 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 8.1302e-05 s, 6.3 MB/s 2024-04-25T09:22:51.428 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T09:22:51.484 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_2 2024-04-25T09:22:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T09:22:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T09:22:51.539 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-25T09:22:51.539 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T09:22:51.539 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T09:22:51.539 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-25 09:22:32.973972129 +0000 2024-04-25T09:22:51.539 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-25 09:20:13.747537322 +0000 2024-04-25T09:22:51.539 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-25 09:20:13.747537322 +0000 2024-04-25T09:22:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-04-25 09:20:13.747537322 +0000 2024-04-25T09:22:51.540 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T09:22:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-25T09:22:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-25T09:22:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000101911 s, 5.0 MB/s 2024-04-25T09:22:51.602 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T09:22:51.658 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_3 2024-04-25T09:22:51.714 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T09:22:51.714 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T09:22:51.714 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T09:22:51.714 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T09:22:51.714 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T09:22:51.714 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-25 09:22:32.974972116 +0000 2024-04-25T09:22:51.714 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-25 09:20:14.078533069 +0000 2024-04-25T09:22:51.714 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-25 09:20:14.078533069 +0000 2024-04-25T09:22:51.715 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-04-25 09:20:14.078533069 +0000 2024-04-25T09:22:51.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T09:22:51.777 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-25T09:22:51.777 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-25T09:22:51.777 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.00010169 s, 5.0 MB/s 2024-04-25T09:22:51.778 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T09:22:51.835 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_4 2024-04-25T09:22:51.891 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T09:22:51.891 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T09:22:51.891 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T09:22:51.891 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T09:22:51.891 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T09:22:51.891 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-25 09:22:32.974972116 +0000 2024-04-25T09:22:51.891 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-25 09:20:14.435528482 +0000 2024-04-25T09:22:51.891 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-25 09:20:14.435528482 +0000 2024-04-25T09:22:51.891 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-04-25 09:20:14.435528482 +0000 2024-04-25T09:22:51.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T09:22:51.953 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-25T09:22:51.953 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-25T09:22:51.953 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000100829 s, 5.1 MB/s 2024-04-25T09:22:51.955 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T09:22:52.011 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-25T09:22:52.011 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:22:52.011 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T09:22:52.029 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T09:22:52.029 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2024-04-25T09:22:52.087 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T09:22:52.087 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T09:22:52.087 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-25T09:22:52.087 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T09:22:52.087 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T09:22:52.087 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-04-25 09:22:21.300620638 +0000 2024-04-25T09:22:52.087 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-04-25 09:20:09.213069108 +0000 2024-04-25T09:22:52.087 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-04-25 09:20:09.213069108 +0000 2024-04-25T09:22:52.087 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-04-25 09:20:09.213069108 +0000 2024-04-25T09:22:52.088 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T09:22:52.152 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-04-25T09:22:52.153 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-04-25T09:22:52.153 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000220145 s, 2.3 MB/s 2024-04-25T09:22:52.154 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T09:22:52.211 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2024-04-25T09:22:52.267 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T09:22:52.268 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T09:22:52.268 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-25T09:22:52.268 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T09:22:52.268 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T09:22:52.268 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-04-25 09:22:21.300620638 +0000 2024-04-25T09:22:52.268 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-04-25 09:20:09.568062958 +0000 2024-04-25T09:22:52.268 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-04-25 09:20:09.568062958 +0000 2024-04-25T09:22:52.268 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-04-25 09:20:09.568062958 +0000 2024-04-25T09:22:52.268 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T09:22:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-04-25T09:22:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-04-25T09:22:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000213936 s, 2.4 MB/s 2024-04-25T09:22:52.334 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T09:22:52.391 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2024-04-25T09:22:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T09:22:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T09:22:52.447 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-25T09:22:52.447 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T09:22:52.447 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T09:22:52.447 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-04-25 09:22:21.300620638 +0000 2024-04-25T09:22:52.447 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-04-25 09:20:09.907057086 +0000 2024-04-25T09:22:52.447 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-04-25 09:20:09.907057086 +0000 2024-04-25T09:22:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-04-25 09:20:09.907057086 +0000 2024-04-25T09:22:52.448 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T09:22:52.510 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-04-25T09:22:52.510 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-04-25T09:22:52.510 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000193873 s, 2.6 MB/s 2024-04-25T09:22:52.512 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T09:22:52.570 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2024-04-25T09:22:52.625 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T09:22:52.626 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T09:22:52.626 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T09:22:52.626 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T09:22:52.626 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T09:22:52.626 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-04-25 09:22:21.301620622 +0000 2024-04-25T09:22:52.626 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-04-25 09:20:10.243051266 +0000 2024-04-25T09:22:52.626 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-04-25 09:20:10.243051266 +0000 2024-04-25T09:22:52.626 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-04-25 09:20:10.243051266 +0000 2024-04-25T09:22:52.626 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T09:22:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-04-25T09:22:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-04-25T09:22:52.689 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000196885 s, 2.6 MB/s 2024-04-25T09:22:52.690 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T09:22:52.746 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-25T09:22:52.746 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-25T09:22:52.746 INFO:tasks.ceph:Generating config... 2024-04-25T09:22:52.747 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-25T09:22:52.747 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-25T09:22:52.747 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-25T09:22:52.747 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-25T09:22:52.747 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-25T09:22:52.747 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T09:22:52.747 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T09:22:52.748 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T09:22:52.748 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-25T09:22:52.748 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-25T09:22:52.748 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-25T09:22:52.748 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-25T09:22:52.748 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T09:22:52.748 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T09:22:52.748 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-25T09:22:52.748 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-25T09:22:52.748 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-25T09:22:52.748 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-25T09:22:52.748 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-25T09:22:52.748 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-25T09:22:52.749 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-25T09:22:52.749 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-25T09:22:52.749 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T09:22:52.749 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T09:22:52.749 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-25T09:22:52.749 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-25T09:22:52.749 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-25T09:22:52.749 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-25T09:22:52.749 INFO:tasks.ceph:Setting up mon.a... 2024-04-25T09:22:52.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-25T09:22:52.818 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/ceph.keyring 2024-04-25T09:22:52.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-25T09:22:52.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T09:22:52.903 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.43')] 2024-04-25T09:22:52.903 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.43', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-25T09:22:52.904 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:22:52.904 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T09:22:52.959 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.43 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-25T09:22:53.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:22:53.064+0000 7fa94e044f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:22:53.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:22:53.064+0000 7fa94e044f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:22:53.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:22:53.065+0000 7fa94e044f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:22:53.051 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-25T09:22:53.052 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: generated fsid 827f1403-0dbf-4d12-8151-d66aeea7580e 2024-04-25T09:22:53.052 INFO:teuthology.orchestra.run.smithi043.stdout:setting min_mon_release = quincy 2024-04-25T09:22:53.052 INFO:teuthology.orchestra.run.smithi043.stdout:epoch 0 2024-04-25T09:22:53.052 INFO:teuthology.orchestra.run.smithi043.stdout:fsid 827f1403-0dbf-4d12-8151-d66aeea7580e 2024-04-25T09:22:53.052 INFO:teuthology.orchestra.run.smithi043.stdout:last_changed 2024-04-25T09:22:53.066414+0000 2024-04-25T09:22:53.052 INFO:teuthology.orchestra.run.smithi043.stdout:created 2024-04-25T09:22:53.066414+0000 2024-04-25T09:22:53.052 INFO:teuthology.orchestra.run.smithi043.stdout:min_mon_release 17 (quincy) 2024-04-25T09:22:53.052 INFO:teuthology.orchestra.run.smithi043.stdout:election_strategy: 1 2024-04-25T09:22:53.052 INFO:teuthology.orchestra.run.smithi043.stdout:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-25T09:22:53.052 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-25T09:22:53.053 DEBUG:teuthology.orchestra.run.smithi043:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T09:22:53.069 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 827f1403-0dbf-4d12-8151-d66aeea7580e... 2024-04-25T09:22:53.070 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T09:22:53.111 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T09:22:53.150 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: chdir = "" 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: auth supported = cephx 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: filestore xattr use omap = true 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: mon clock drift allowed = 1.000 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: osd crush chooseleaf type = 0 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: auth debug = true 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on old message = true 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on bug = true 2024-04-25T09:22:53.151 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: mon pg warn max object skew = 0 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default size = 2 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow primary affinity = true 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow pg remap = true 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on legacy crush tunables = false 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on crush straw calc version zero = false 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on no sortbitwise = false 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on osd down out interval zero = false 2024-04-25T09:22:53.152 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on too few osds = false 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: mon_allow_pool_size_one = true 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: osd default data pool replay window = 5 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: mon allow pool delete = true 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: mon cluster log file level = debug 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: debug asserts on shutdown = true 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: mon health detail to clog = false 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: mon host = 172.21.15.43 2024-04-25T09:22:53.153 INFO:teuthology.orchestra.run.smithi043.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout: ms client mode = crc 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout: ms cluster mode = crc 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout: ms mon client mode = crc 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout: ms mon cluster mode = crc 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout: ms mon service mode = crc 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout: ms service mode = crc 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout: fsid = 827f1403-0dbf-4d12-8151-d66aeea7580e 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout:[osd] 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout: osd journal size = 100 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub load threshold = 5.0 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub max interval = 600 2024-04-25T09:22:53.154 INFO:teuthology.orchestra.run.smithi043.stdout: osd mclock profile = high_recovery_ops 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: osd recover clone overlap = true 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: osd recovery max chunk = 1048576 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug shutdown = true 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug op order = true 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug verify stray on activate = true 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug trim objects = true 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: osd open classes on start = true 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug pg log writeout = true 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: osd deep scrub update digest min age = 30 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: osd map max advance = 10 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.155 INFO:teuthology.orchestra.run.smithi043.stdout: journal zero on create = true 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: filestore ondisk finisher threads = 3 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: filestore apply finisher threads = 3 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: bdev debug aio = true 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug misdirected ops = true 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: debug monc = 20 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: crimson alien thread cpu cores = 6-7 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: crimson osd obc lru size = 10 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: debug osd = 20 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: osd shutdown pgref assert = True 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr] 2024-04-25T09:22:53.156 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: debug mgr = 20 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: mgr/telemetry/nag = false 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout:[mon] 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: debug paxos = 20 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: mon data avail warn = 5 2024-04-25T09:22:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: mon mgr mkfs grace = 240 2024-04-25T09:22:53.158 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2024-04-25T09:22:53.158 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd reporter subtree level = osd 2024-04-25T09:22:53.158 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd prime pg temp = true 2024-04-25T09:22:53.158 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2024-04-25T09:22:53.158 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.158 INFO:teuthology.orchestra.run.smithi043.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T09:22:53.158 INFO:teuthology.orchestra.run.smithi043.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T09:22:53.158 INFO:teuthology.orchestra.run.smithi043.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T09:22:53.158 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.158 INFO:teuthology.orchestra.run.smithi043.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T09:22:53.159 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T09:22:53.159 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T09:22:53.159 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.159 INFO:teuthology.orchestra.run.smithi043.stdout: # 1m isn't quite enough 2024-04-25T09:22:53.159 INFO:teuthology.orchestra.run.smithi043.stdout: mon_down_mkfs_grace = 2m 2024-04-25T09:22:53.159 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.159 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_filestore_osds = false 2024-04-25T09:22:53.159 INFO:teuthology.orchestra.run.smithi043.stdout: mon min osdmap epochs = 50 2024-04-25T09:22:53.159 INFO:teuthology.orchestra.run.smithi043.stdout: mon osdmap full prune interval = 2 2024-04-25T09:22:53.159 INFO:teuthology.orchestra.run.smithi043.stdout: mon osdmap full prune min = 15 2024-04-25T09:22:53.159 INFO:teuthology.orchestra.run.smithi043.stdout: mon osdmap full prune txsize = 2 2024-04-25T09:22:53.159 INFO:teuthology.orchestra.run.smithi043.stdout: paxos service trim min = 10 2024-04-25T09:22:53.159 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default crimson = True 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout:[client] 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout: rgw cache enabled = true 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable ops log = true 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable usage log = true 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.a] 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.0] 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout: crimson seastar cpu cores = 0-2 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.1] 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout: crimson seastar cpu cores = 3-5 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.2] 2024-04-25T09:22:53.160 INFO:teuthology.orchestra.run.smithi043.stdout: crimson seastar cpu cores = 0-2 2024-04-25T09:22:53.161 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.3] 2024-04-25T09:22:53.161 INFO:teuthology.orchestra.run.smithi043.stdout: crimson seastar cpu cores = 3-5 2024-04-25T09:22:53.161 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-04-25T09:22:53.161 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-04-25T09:22:53.161 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T09:22:53.161 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.162 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T09:22:53.163 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = 172.21.15.43 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: ms client mode = crc 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: ms cluster mode = crc 2024-04-25T09:22:53.164 INFO:teuthology.orchestra.run.smithi164.stdout: ms mon client mode = crc 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: ms mon cluster mode = crc 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: ms mon service mode = crc 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: ms service mode = crc 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 827f1403-0dbf-4d12-8151-d66aeea7580e 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-04-25T09:22:53.165 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-04-25T09:22:53.166 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout: debug monc = 20 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout: crimson alien thread cpu cores = 6-7 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout: crimson osd obc lru size = 10 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout: osd shutdown pgref assert = True 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-04-25T09:22:53.167 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-04-25T09:22:53.168 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-04-25T09:22:53.169 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout: mon min osdmap epochs = 50 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout: mon osdmap full prune interval = 2 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout: mon osdmap full prune min = 15 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout: mon osdmap full prune txsize = 2 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout: paxos service trim min = 10 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default crimson = True 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T09:22:53.170 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-04-25T09:22:53.171 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.0] 2024-04-25T09:22:53.171 INFO:teuthology.orchestra.run.smithi164.stdout: crimson seastar cpu cores = 0-2 2024-04-25T09:22:53.171 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.1] 2024-04-25T09:22:53.171 INFO:teuthology.orchestra.run.smithi164.stdout: crimson seastar cpu cores = 3-5 2024-04-25T09:22:53.171 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.2] 2024-04-25T09:22:53.171 INFO:teuthology.orchestra.run.smithi164.stdout: crimson seastar cpu cores = 0-2 2024-04-25T09:22:53.171 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.3] 2024-04-25T09:22:53.171 INFO:teuthology.orchestra.run.smithi164.stdout: crimson seastar cpu cores = 3-5 2024-04-25T09:22:53.172 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-25T09:22:53.173 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-25T09:22:53.275 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T09:22:53.276 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:22:53.276 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-25T09:22:53.291 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:22:53.291 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-25T09:22:53.346 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi043.front.sepia.ceph.com 2024-04-25T09:22:53.346 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:22:53.346 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T09:22:53.346 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T09:22:53.422 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:22:53.422 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T09:22:53.477 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2024-04-25T09:22:53.478 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:22:53.478 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T09:22:53.478 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T09:22:53.514 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:22:53.514 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T09:22:53.570 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T09:22:53.570 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T09:22:53.571 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T09:22:53.676 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T09:22:53.679 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T09:22:53.679 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T09:22:53.680 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T09:22:53.729 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-25T09:22:53.741 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-25T09:22:53.792 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-25T09:22:53.804 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T09:22:53.805 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-25T09:22:53.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-25T09:22:53.829 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-25T09:22:53.829 INFO:tasks.ceph:role: osd.0 2024-04-25T09:22:53.829 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-25T09:22:53.829 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T09:22:53.892 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T09:22:53.893 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T09:22:53.893 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T09:22:53.893 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T09:22:53.893 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T09:22:53.893 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-04-25T09:22:53.893 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T09:22:53.893 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T09:22:53.893 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T09:22:53.893 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T09:22:53.925 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-04-25T09:22:53.927 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-04-25T09:22:53.927 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-25T09:22:53.996 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-25T09:22:54.068 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-25T09:22:54.133 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-25T09:22:54.133 INFO:tasks.ceph:role: osd.1 2024-04-25T09:22:54.133 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-25T09:22:54.133 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T09:22:54.198 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T09:22:54.198 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T09:22:54.198 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T09:22:54.198 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T09:22:54.198 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T09:22:54.198 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-04-25T09:22:54.198 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T09:22:54.198 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T09:22:54.198 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T09:22:54.199 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T09:22:54.230 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-04-25T09:22:54.232 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-04-25T09:22:54.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-25T09:22:54.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-25T09:22:54.369 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T09:22:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-25T09:22:54.678 INFO:teuthology.orchestra.run.smithi043.stderr:INFO 2024-04-25 09:22:54,693 seastar - Reactor backend: linux-aio 2024-04-25T09:22:54.690 INFO:teuthology.orchestra.run.smithi043.stderr:==23954==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-25T09:22:54.806 INFO:teuthology.orchestra.run.smithi043.stderr:INFO 2024-04-25 09:22:54,821 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-25T09:22:54.806 INFO:teuthology.orchestra.run.smithi043.stderr:INFO 2024-04-25 09:22:54,821 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-25T09:22:54.807 INFO:teuthology.orchestra.run.smithi043.stderr:DEBUG 2024-04-25 09:22:54,822 [shard 0:main] seastar - reactor::drain 2024-04-25T09:22:55.363 INFO:teuthology.orchestra.run.smithi043.stderr:----------------------------------------------------- 2024-04-25T09:22:55.363 INFO:teuthology.orchestra.run.smithi043.stderr:Suppressions used: 2024-04-25T09:22:55.363 INFO:teuthology.orchestra.run.smithi043.stderr: count bytes template 2024-04-25T09:22:55.364 INFO:teuthology.orchestra.run.smithi043.stderr: 1 8 MallocExtension::Register 2024-04-25T09:22:55.364 INFO:teuthology.orchestra.run.smithi043.stderr:----------------------------------------------------- 2024-04-25T09:22:55.364 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-25T09:22:55.418 INFO:teuthology.orchestra.run.smithi043.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-25T09:22:55.422 INFO:teuthology.orchestra.run.smithi043.stderr:INFO 2024-04-25 09:22:55,437 seastar - Reactor backend: linux-aio 2024-04-25T09:22:55.431 INFO:teuthology.orchestra.run.smithi043.stderr:==23952==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-25T09:22:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9f08b9 0xb8c3380 0xb8c48fc 0xb8c4b12 0xb8c4c68 0xb8c5131 0x3e6ef 0x113cb0 0x4a0bd 0x35b7c 0x3629e 0x5df5a44 0x5df5c0c 0x5df5e3d 0x5df6113 0x5e44953 0x5e44b59 0x5dc3640 0x5dc381a 0x5dc3da6 0x5c088dd 0x5c239a1 0x6efc314 0x6f0dae7 0x6f0ef84 0x6f0f009 0x6f0f10b 0x6f0f203 0x6f0f410 0x6f0f5f6 0x6f0f945 0x6f0fce9 0x6f0ff19 0x6f10183 0x6f10450 0x6f1066e 0x6f10726 0x6f107ec 0x6f108cd 0x5a711d6 0x5a712e0 0x5a7139c 0x5a71458 0x5a71514 0x5a71661 0x5a717ec 0x5a71bad 0x5a8daee 0xb8d6005 0xb8f029f 0xb99129d 0xb9928eb 0xb66f0d1 0xb66fa4d 0x368758a 0x2958f 0x2963f 0x3473444 2024-04-25T09:22:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:kernel callstack: 2024-04-25T09:22:58.891 INFO:teuthology.orchestra.run.smithi043.stderr:----------------------------------------------------- 2024-04-25T09:22:58.891 INFO:teuthology.orchestra.run.smithi043.stderr:Suppressions used: 2024-04-25T09:22:58.892 INFO:teuthology.orchestra.run.smithi043.stderr: count bytes template 2024-04-25T09:22:58.892 INFO:teuthology.orchestra.run.smithi043.stderr: 1 8 MallocExtension::Register 2024-04-25T09:22:58.892 INFO:teuthology.orchestra.run.smithi043.stderr:----------------------------------------------------- 2024-04-25T09:22:58.892 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-25T09:22:58.892 INFO:teuthology.orchestra.run.smithi043.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 827f1403-0dbf-4d12-8151-d66aeea7580e 2024-04-25T09:22:58.919 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-25T09:22:58.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T09:22:59.249 INFO:teuthology.orchestra.run.smithi043.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-25T09:22:59.250 INFO:teuthology.orchestra.run.smithi043.stderr:INFO 2024-04-25 09:22:59,265 seastar - Reactor backend: linux-aio 2024-04-25T09:22:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:==24763==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-25T09:22:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9f08b9 0xb8c3380 0xb8c48fc 0xb8c4b12 0xb8c4c68 0xb8c5131 0x3e6ef 0xdfde4 0x349fdf3 0x349fffc 0x34cd66d 0x36bb0ee 0x3ff30e5 0x5dc361e 0x5dc3b74 0x5c088dd 0x5c239a1 0x6efc314 0x6f0dae7 0x6f0ef84 0x6f0f009 0x6f0f10b 0x6f0f203 0x6f0f410 0x6f0f5f6 0x6f0f945 0x6f0fce9 0x6f0ff19 0x6f10183 0x6f10450 0x6f1066e 0x6f10726 0x6f107ec 0x6f108cd 0x5a711d6 0x5a712e0 0x5a7139c 0x5a71458 0x5a71514 0x5a71661 0x5a717ec 0x5a71bad 0x5a8daee 0xb8d6005 0xb8f029f 0xb99129d 0xb9928eb 0xb66f0d1 0xb66fa4d 0x37e691a 0x37e7473 0x3686d7a 0x2958f 0x2963f 0x3473444 2024-04-25T09:22:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:kernel callstack: 2024-04-25T09:22:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:INFO 2024-04-25 09:22:59,387 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-25T09:22:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:INFO 2024-04-25 09:22:59,387 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-25T09:22:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:DEBUG 2024-04-25 09:22:59,388 [shard 0:main] seastar - reactor::drain 2024-04-25T09:22:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:----------------------------------------------------- 2024-04-25T09:22:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:Suppressions used: 2024-04-25T09:22:59.930 INFO:teuthology.orchestra.run.smithi043.stderr: count bytes template 2024-04-25T09:22:59.930 INFO:teuthology.orchestra.run.smithi043.stderr: 1 8 MallocExtension::Register 2024-04-25T09:22:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:----------------------------------------------------- 2024-04-25T09:22:59.930 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-25T09:22:59.986 INFO:teuthology.orchestra.run.smithi043.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-25T09:22:59.990 INFO:teuthology.orchestra.run.smithi043.stderr:INFO 2024-04-25 09:23:00,005 seastar - Reactor backend: linux-aio 2024-04-25T09:22:59.999 INFO:teuthology.orchestra.run.smithi043.stderr:==24761==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-25T09:23:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9f08b9 0xb8c3380 0xb8c48fc 0xb8c4b12 0xb8c4c68 0xb8c5131 0x3e6ef 0x3635e 0x5df5a44 0x5df5c5a 0x5df5e3d 0x5df6113 0x5e44953 0x5e44b59 0x5dc3640 0x5dc381a 0x5dc3da6 0x5c088dd 0x5c239a1 0x6efc314 0x6f0dae7 0x6f0ef84 0x6f0f009 0x6f0f10b 0x6f0f203 0x6f0f410 0x6f0f5f6 0x6f0f945 0x6f0fce9 0x6f0ff19 0x6f10183 0x6f10450 0x6f1066e 0x6f10726 0x6f107ec 0x6f108cd 0x5a711d6 0x5a712e0 0x5a7139c 0x5a71458 0x5a71514 0x5a71661 0x5a717ec 0x5a71bad 0x5a8daee 0xb8d6005 0xb8f029f 0xb99129d 0xb9928eb 0xb66f0d1 0xb66fa4d 0x368758a 0x2958f 0x2963f 0x3473444 2024-04-25T09:23:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:kernel callstack: 2024-04-25T09:23:03.502 INFO:teuthology.orchestra.run.smithi043.stderr:----------------------------------------------------- 2024-04-25T09:23:03.502 INFO:teuthology.orchestra.run.smithi043.stderr:Suppressions used: 2024-04-25T09:23:03.502 INFO:teuthology.orchestra.run.smithi043.stderr: count bytes template 2024-04-25T09:23:03.502 INFO:teuthology.orchestra.run.smithi043.stderr: 1 8 MallocExtension::Register 2024-04-25T09:23:03.502 INFO:teuthology.orchestra.run.smithi043.stderr:----------------------------------------------------- 2024-04-25T09:23:03.502 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-25T09:23:03.502 INFO:teuthology.orchestra.run.smithi043.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 827f1403-0dbf-4d12-8151-d66aeea7580e 2024-04-25T09:23:03.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-25T09:23:03.553 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-25T09:23:03.582 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-25T09:23:03.582 INFO:tasks.ceph:role: osd.2 2024-04-25T09:23:03.582 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com 2024-04-25T09:23:03.582 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T09:23:03.655 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T09:23:03.656 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T09:23:03.656 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T09:23:03.656 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T09:23:03.656 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T09:23:03.656 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-04-25T09:23:03.656 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T09:23:03.656 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T09:23:03.656 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T09:23:03.656 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T09:23:04.171 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-04-25T09:23:04.174 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-04-25T09:23:04.174 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-04-25T09:23:04.327 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-25T09:23:04.360 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-25T09:23:04.427 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-25T09:23:04.427 INFO:tasks.ceph:role: osd.3 2024-04-25T09:23:04.427 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com 2024-04-25T09:23:04.427 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T09:23:04.500 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T09:23:04.500 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T09:23:04.500 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T09:23:04.500 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T09:23:04.500 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T09:23:04.500 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-04-25T09:23:04.501 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T09:23:04.501 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T09:23:04.501 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T09:23:04.501 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T09:23:04.942 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-04-25T09:23:04.945 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-04-25T09:23:04.946 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-04-25T09:23:05.040 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-25T09:23:05.067 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T09:23:05.387 INFO:teuthology.orchestra.run.smithi164.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-25T09:23:05.388 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2024-04-25 09:23:05,403 seastar - Reactor backend: linux-aio 2024-04-25T09:23:05.407 INFO:teuthology.orchestra.run.smithi164.stderr:==23772==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-25T09:23:05.519 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2024-04-25 09:23:05,534 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-25T09:23:05.519 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2024-04-25 09:23:05,534 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-25T09:23:05.519 INFO:teuthology.orchestra.run.smithi164.stderr:DEBUG 2024-04-25 09:23:05,535 [shard 0:main] seastar - reactor::drain 2024-04-25T09:23:06.067 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2024-04-25T09:23:06.067 INFO:teuthology.orchestra.run.smithi164.stderr:Suppressions used: 2024-04-25T09:23:06.067 INFO:teuthology.orchestra.run.smithi164.stderr: count bytes template 2024-04-25T09:23:06.067 INFO:teuthology.orchestra.run.smithi164.stderr: 1 8 MallocExtension::Register 2024-04-25T09:23:06.067 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2024-04-25T09:23:06.068 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-04-25T09:23:06.126 INFO:teuthology.orchestra.run.smithi164.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-25T09:23:06.129 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2024-04-25 09:23:06,144 seastar - Reactor backend: linux-aio 2024-04-25T09:23:06.139 INFO:teuthology.orchestra.run.smithi164.stderr:==23770==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-25T09:23:09.557 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2024-04-25T09:23:09.557 INFO:teuthology.orchestra.run.smithi164.stderr:Suppressions used: 2024-04-25T09:23:09.557 INFO:teuthology.orchestra.run.smithi164.stderr: count bytes template 2024-04-25T09:23:09.557 INFO:teuthology.orchestra.run.smithi164.stderr: 1 8 MallocExtension::Register 2024-04-25T09:23:09.557 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2024-04-25T09:23:09.557 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-04-25T09:23:09.557 INFO:teuthology.orchestra.run.smithi164.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 827f1403-0dbf-4d12-8151-d66aeea7580e 2024-04-25T09:23:09.582 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-25T09:23:09.606 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T09:23:09.888 INFO:teuthology.orchestra.run.smithi164.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-25T09:23:09.889 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2024-04-25 09:23:09,904 seastar - Reactor backend: linux-aio 2024-04-25T09:23:09.894 INFO:teuthology.orchestra.run.smithi164.stderr:==24581==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-25T09:23:09.998 INFO:teuthology.orchestra.run.smithi164.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9f08b9 0xb8c3380 0xb8c48fc 0xb8c4b12 0xb8c4c68 0xb8c5131 0x3e6ef 0x37b44f8 0x37b483a 0x37650e7 0x37653af 0x5c15616 0x5c19fb9 0x5c1f226 0x5c1f828 0x6efc383 0x6f0dae7 0x6f0ef84 0x6f0f009 0x6f0f10b 0x6f0f203 0x6f0f410 0x6f0f5f6 0x6f0f945 0x6f0fce9 0x6f0ff19 0x6f10183 0x6f10450 0x6f1066e 0x6f10726 0x6f107ec 0x6f108cd 0x5a711d6 0x5a712e0 0x5a7139c 0x5a71458 0x5a71514 0x5a71661 0x5a717ec 0x5a71bad 0x5a8daee 0xb8d6005 0xb8f029f 0xb99129d 0xb9928eb 0xb66f0d1 0xb66fa4d 0x37e691a 0x37e7473 0x3686d7a 0x2958f 0x2963f 0x3473444 2024-04-25T09:23:09.998 INFO:teuthology.orchestra.run.smithi164.stderr:kernel callstack: 2024-04-25T09:23:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2024-04-25 09:23:10,018 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-25T09:23:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2024-04-25 09:23:10,018 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-25T09:23:10.003 INFO:teuthology.orchestra.run.smithi164.stderr:DEBUG 2024-04-25 09:23:10,019 [shard 0:main] seastar - reactor::drain 2024-04-25T09:23:10.533 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2024-04-25T09:23:10.533 INFO:teuthology.orchestra.run.smithi164.stderr:Suppressions used: 2024-04-25T09:23:10.533 INFO:teuthology.orchestra.run.smithi164.stderr: count bytes template 2024-04-25T09:23:10.533 INFO:teuthology.orchestra.run.smithi164.stderr: 1 8 MallocExtension::Register 2024-04-25T09:23:10.533 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2024-04-25T09:23:10.533 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-04-25T09:23:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-25T09:23:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2024-04-25 09:23:10,611 seastar - Reactor backend: linux-aio 2024-04-25T09:23:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:==24579==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-25T09:23:14.058 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2024-04-25T09:23:14.058 INFO:teuthology.orchestra.run.smithi164.stderr:Suppressions used: 2024-04-25T09:23:14.058 INFO:teuthology.orchestra.run.smithi164.stderr: count bytes template 2024-04-25T09:23:14.058 INFO:teuthology.orchestra.run.smithi164.stderr: 1 8 MallocExtension::Register 2024-04-25T09:23:14.058 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2024-04-25T09:23:14.058 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-04-25T09:23:14.059 INFO:teuthology.orchestra.run.smithi164.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid 827f1403-0dbf-4d12-8151-d66aeea7580e 2024-04-25T09:23:14.078 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-25T09:23:14.102 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T09:23:14.102 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:23:14.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-25T09:23:14.134 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:23:14.134 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-25T09:23:14.202 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:23:14.203 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-25T09:23:14.226 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:23:14.226 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-25T09:23:14.289 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:23:14.289 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-25T09:23:14.352 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:23:14.353 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-25T09:23:14.372 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:23:14.372 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-25T09:23:14.408 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T09:23:14.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T09:23:14.439 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.0] 2024-04-25T09:23:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDvICpmM6XsIhAAUuKgZAK3r5QyCMNTV2Mf5g== 2024-04-25T09:23:14.440 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.1] 2024-04-25T09:23:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQD0ICpmaqtRChAAEgHNCNjvmNeUlbQxDnoMVg== 2024-04-25T09:23:14.440 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr.x] 2024-04-25T09:23:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDtICpmxvY7KRAAmwpC8G4P269G5w5OaPMlLA== 2024-04-25T09:23:14.440 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.2] 2024-04-25T09:23:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQD6ICpm4Q0WERAAerwM1IgjCduA+IcCofoGcQ== 2024-04-25T09:23:14.440 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.3] 2024-04-25T09:23:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQD+ICpm594GLRAA0m3hXqmkEZvMff0C3YpHLA== 2024-04-25T09:23:14.440 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-04-25T09:23:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDtICpmNnBeLBAAUalu8dYM5DSb7LCbsh59fw== 2024-04-25T09:23:14.440 INFO:teuthology.orchestra.run.smithi043.stdout:[client.1] 2024-04-25T09:23:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDtICpmCxMeMBAAefDPUnQvzcPPe1nlpLNKug== 2024-04-25T09:23:14.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T09:23:14.547 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T09:23:14.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T09:23:14.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T09:23:14.732 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T09:23:14.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T09:23:14.857 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T09:23:14.920 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T09:23:14.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-25T09:23:14.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T09:23:15.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:15.074+0000 7fb556a8ca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:15.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:15.075+0000 7fb556a8ca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:15.077 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-25T09:23:15.102 DEBUG:teuthology.orchestra.run.smithi043:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T09:23:15.157 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-25T09:23:15.158 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-25T09:23:15.158 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-25T09:23:15.200 INFO:tasks.ceph.mon.a:Started 2024-04-25T09:23:15.200 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-25T09:23:15.201 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-25T09:23:15.201 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-25T09:23:15.203 INFO:tasks.ceph.mgr.x:Started 2024-04-25T09:23:15.204 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T09:23:15.204 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd crush tunables default 2024-04-25T09:23:15.283 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-25T09:23:15.296+0000 7f311048ca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:15.283 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-25T09:23:15.296+0000 7f311048ca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:15.283 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-25T09:23:15.297+0000 7f311048ca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:15.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:15.320+0000 7f89b0574000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:15.325+0000 7f818cc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:15.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:15.332+0000 7f89b0574000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:15.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:15.333+0000 7f89b0574000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:15.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:15.336+0000 7f818cc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:adjusted tunables profile to default 2024-04-25T09:23:15.409 INFO:tasks.ceph:check_enable_crimson: True 2024-04-25T09:23:15.409 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-25T09:23:15.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-25T09:23:15.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:15.435+0000 7f89b0574000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T09:23:15.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:15.526+0000 7f74e5aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:15.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:15.536+0000 7f74e5aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:15.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:15.584+0000 7f89b0574000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T09:23:15.606 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-25T09:23:15.620+0000 7f31083f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-25T09:23:15.606 INFO:tasks.ceph.mon.a.smithi043.stderr:Please be aware that this feature is experimental, untested, 2024-04-25T09:23:15.606 INFO:tasks.ceph.mon.a.smithi043.stderr:unsupported, and may result in data corruption, data loss, 2024-04-25T09:23:15.606 INFO:tasks.ceph.mon.a.smithi043.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-25T09:23:15.606 INFO:tasks.ceph.mon.a.smithi043.stderr:feature with important data. 2024-04-25T09:23:15.606 INFO:tasks.ceph.mon.a.smithi043.stderr: 2024-04-25T09:23:15.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:15.636+0000 7f89b0574000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T09:23:15.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:15.894+0000 7f89b0574000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T09:23:15.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:15.940+0000 7f89b0574000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T09:23:16.412 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-25T09:23:16.412 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:23:16.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-25T09:23:16.439 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:23:16.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-25T09:23:16.506 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:23:16.506 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-25T09:23:16.530 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-04-25T09:23:16.530 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-25T09:23:16.594 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 2376fa93-8105-45bc-95ff-354e31340c3f 0 2024-04-25T09:23:16.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:16.632+0000 7f89b0574000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T09:23:16.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:16.726+0000 7f89b0574000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T09:23:16.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-25T09:23:16.733+0000 7f878ad5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:16.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-25T09:23:16.742+0000 7f878ad5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:16.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:16.771+0000 7f89b0574000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T09:23:16.808 INFO:teuthology.orchestra.run.smithi164.stdout:0 2024-04-25T09:23:16.820 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new f2c54475-fe0d-4041-b9d2-a18634c6da39 1 2024-04-25T09:23:16.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-25T09:23:16.933+0000 7f5a9486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:16.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-25T09:23:16.942+0000 7f5a9486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:16.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:16.946+0000 7f89b0574000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T09:23:16.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:17.001+0000 7f89b0574000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T09:23:17.008 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-04-25T09:23:17.020 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 01b299ea-0a8e-49e4-bbc9-edacd267f29d 2 2024-04-25T09:23:17.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:17.057+0000 7f89b0574000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T09:23:17.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-25T09:23:17.134+0000 7f38656ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:17.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-25T09:23:17.143+0000 7f38656ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:17.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:17.176+0000 7f89b0574000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T09:23:17.208 INFO:teuthology.orchestra.run.smithi164.stdout:2 2024-04-25T09:23:17.219 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 40f3ae87-d337-4a5b-a8b2-6f44aaca954e 3 2024-04-25T09:23:17.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-25T09:23:17.375+0000 7f7d4b19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:17.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-04-25T09:23:17.384+0000 7f7d4b19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:17.449 INFO:teuthology.orchestra.run.smithi164.stdout:3 2024-04-25T09:23:17.461 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-25T09:23:17.495 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-25T09:23:17.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-25T09:23:17.539 INFO:tasks.ceph.osd.0:Started 2024-04-25T09:23:17.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-25T09:23:17.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:17.570+0000 7f89b0574000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T09:23:17.572 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-25T09:23:17.573 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-25T09:23:17.615 INFO:tasks.ceph.osd.1:Started 2024-04-25T09:23:17.615 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-25T09:23:17.644 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-25T09:23:17.645 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-25T09:23:17.687 INFO:tasks.ceph.osd.2:Started 2024-04-25T09:23:17.698 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-04-25T09:23:17.726 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-25T09:23:17.727 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-25T09:23:17.769 INFO:tasks.ceph.osd.3:Started 2024-04-25T09:23:17.770 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T09:23:17.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:17.844+0000 7f89b0574000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T09:23:17.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:17.896+0000 7fbd794ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:17.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:17.904+0000 7fbd794ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:17.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:17.905+0000 7f89b0574000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T09:23:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:23:17.962 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":7,"fsid":"827f1403-0dbf-4d12-8151-d66aeea7580e","created":"2024-04-25T09:23:15.321458+0000","modified":"2024-04-25T09:23:17.464337+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T09:23:17.464337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2376fa93-8105-45bc-95ff-354e31340c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f2c54475-fe0d-4041-b9d2-a18634c6da39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"01b299ea-0a8e-49e4-bbc9-edacd267f29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"40f3ae87-d337-4a5b-a8b2-6f44aaca954e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T09:23:17.973 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-25T09:23:17.973 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T09:23:18.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:18.034+0000 7f89b0574000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T09:23:18.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:18.324+0000 7f89b0574000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T09:23:18.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:18.376+0000 7f89b0574000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T09:23:18.375 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T09:23:18.481 INFO:teuthology.misc.health.smithi043.stderr:2024-04-25T09:23:18.491+0000 7f79068ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:18.487 INFO:teuthology.misc.health.smithi043.stderr:2024-04-25T09:23:18.501+0000 7f79068ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:18.531 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:18.545+0000 7f89b0574000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T09:23:18.557 INFO:teuthology.misc.health.smithi043.stdout: 2024-04-25T09:23:18.557 INFO:teuthology.misc.health.smithi043.stdout:{"epoch":7,"fsid":"827f1403-0dbf-4d12-8151-d66aeea7580e","created":"2024-04-25T09:23:15.321458+0000","modified":"2024-04-25T09:23:17.464337+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T09:23:17.464337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2376fa93-8105-45bc-95ff-354e31340c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f2c54475-fe0d-4041-b9d2-a18634c6da39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"01b299ea-0a8e-49e4-bbc9-edacd267f29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"40f3ae87-d337-4a5b-a8b2-6f44aaca954e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T09:23:18.568 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-04-25T09:23:18.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:18.685+0000 7f89b0574000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T09:23:18.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:18.739+0000 7f89b0574000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T09:23:18.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:18.816+0000 7f89b0574000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T09:23:18.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:18.864+0000 7f89b0574000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T09:23:19.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:19.359+0000 7f89b0574000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T09:23:19.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:19.494+0000 7f89b0574000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T09:23:20.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-04-25T09:23:20.504+0000 7f894f7e8640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-25T09:23:24.972 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T09:23:25.082 INFO:teuthology.misc.health.smithi043.stderr:2024-04-25T09:23:25.091+0000 7f77e76ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:25.086 INFO:teuthology.misc.health.smithi043.stderr:2024-04-25T09:23:25.100+0000 7f77e76ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:25.291 INFO:teuthology.misc.health.smithi043.stdout: 2024-04-25T09:23:25.291 INFO:teuthology.misc.health.smithi043.stdout:{"epoch":11,"fsid":"827f1403-0dbf-4d12-8151-d66aeea7580e","created":"2024-04-25T09:23:15.321458+0000","modified":"2024-04-25T09:23:24.542071+0000","last_up_change":"2024-04-25T09:23:21.536091+0000","last_in_change":"2024-04-25T09:23:17.464337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T09:23:23.521590+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2376fa93-8105-45bc-95ff-354e31340c3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":215454835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":215454835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":215454835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6809","nonce":215454835}]},"public_addr":"172.21.15.43:6801/215454835","cluster_addr":"172.21.15.43:6802/215454835","heartbeat_back_addr":"172.21.15.43:6810/215454835","heartbeat_front_addr":"172.21.15.43:6809/215454835","state":["exists","up"]},{"osd":1,"uuid":"f2c54475-fe0d-4041-b9d2-a18634c6da39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3844277216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":3844277216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":3844277216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3844277216}]},"public_addr":"172.21.15.43:6804/3844277216","cluster_addr":"172.21.15.43:6805/3844277216","heartbeat_back_addr":"172.21.15.43:6811/3844277216","heartbeat_front_addr":"172.21.15.43:6808/3844277216","state":["exists","up"]},{"osd":2,"uuid":"01b299ea-0a8e-49e4-bbc9-edacd267f29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":1388622007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":1388622007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1388622007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":1388622007}]},"public_addr":"172.21.15.164:6804/1388622007","cluster_addr":"172.21.15.164:6805/1388622007","heartbeat_back_addr":"172.21.15.164:6808/1388622007","heartbeat_front_addr":"172.21.15.164:6811/1388622007","state":["exists","up"]},{"osd":3,"uuid":"40f3ae87-d337-4a5b-a8b2-6f44aaca954e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":4179274706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":4179274706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":4179274706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":4179274706}]},"public_addr":"172.21.15.164:6806/4179274706","cluster_addr":"172.21.15.164:6807/4179274706","heartbeat_back_addr":"172.21.15.164:6813/4179274706","heartbeat_front_addr":"172.21.15.164:6814/4179274706","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T09:23:25.304 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-04-25T09:23:25.304 INFO:tasks.ceph:Creating RBD pool 2024-04-25T09:23:25.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-25T09:23:25.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:25.423+0000 7fd5df4ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:25.467+0000 7fd5df4ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:26.541 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'rbd' created 2024-04-25T09:23:26.563 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-25T09:23:26.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:26.695+0000 7f4898e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:26.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:26.710+0000 7f4898e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:27.569 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-25T09:23:27.591 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-25T09:23:27.591 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T09:23:27.592 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-25T09:23:27.592 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-25T09:23:27.592 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T09:23:27.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:27.732+0000 7f99ae0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:27.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:27.740+0000 7f99ae0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:27.977 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:23:27.989 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3137297936},{"type":"v1","addr":"172.21.15.164:6801","nonce":3137297936}]},"active_addr":"172.21.15.164:6801/3137297936","active_change":"2024-04-25T09:23:19.497623+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":400081935}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":313028830}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":3554617072}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":292447314}]}]} 2024-04-25T09:23:27.991 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-25T09:23:27.991 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-25T09:23:27.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T09:23:28.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.112+0000 7f53ac59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.120+0000 7f53ac59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.304 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:23:28.304 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":14,"fsid":"827f1403-0dbf-4d12-8151-d66aeea7580e","created":"2024-04-25T09:23:15.321458+0000","modified":"2024-04-25T09:23:27.582070+0000","last_up_change":"2024-04-25T09:23:21.536091+0000","last_in_change":"2024-04-25T09:23:17.464337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T09:23:23.521590+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T09:23:25.673624+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2376fa93-8105-45bc-95ff-354e31340c3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":215454835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":215454835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":215454835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6809","nonce":215454835}]},"public_addr":"172.21.15.43:6801/215454835","cluster_addr":"172.21.15.43:6802/215454835","heartbeat_back_addr":"172.21.15.43:6810/215454835","heartbeat_front_addr":"172.21.15.43:6809/215454835","state":["exists","up"]},{"osd":1,"uuid":"f2c54475-fe0d-4041-b9d2-a18634c6da39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3844277216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":3844277216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":3844277216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3844277216}]},"public_addr":"172.21.15.43:6804/3844277216","cluster_addr":"172.21.15.43:6805/3844277216","heartbeat_back_addr":"172.21.15.43:6811/3844277216","heartbeat_front_addr":"172.21.15.43:6808/3844277216","state":["exists","up"]},{"osd":2,"uuid":"01b299ea-0a8e-49e4-bbc9-edacd267f29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":1388622007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":1388622007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1388622007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":1388622007}]},"public_addr":"172.21.15.164:6804/1388622007","cluster_addr":"172.21.15.164:6805/1388622007","heartbeat_back_addr":"172.21.15.164:6808/1388622007","heartbeat_front_addr":"172.21.15.164:6811/1388622007","state":["exists","up"]},{"osd":3,"uuid":"40f3ae87-d337-4a5b-a8b2-6f44aaca954e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":4179274706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":4179274706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":4179274706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":4179274706}]},"public_addr":"172.21.15.164:6806/4179274706","cluster_addr":"172.21.15.164:6807/4179274706","heartbeat_back_addr":"172.21.15.164:6813/4179274706","heartbeat_front_addr":"172.21.15.164:6814/4179274706","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T09:23:28.316 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-25T09:23:28.316 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T09:23:28.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.437+0000 7ff6ed8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.445+0000 7ff6ed8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:23:28.630 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":15,"fsid":"827f1403-0dbf-4d12-8151-d66aeea7580e","created":"2024-04-25T09:23:15.321458+0000","modified":"2024-04-25T09:23:28.584648+0000","last_up_change":"2024-04-25T09:23:21.536091+0000","last_in_change":"2024-04-25T09:23:17.464337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T09:23:23.521590+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T09:23:25.673624+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2376fa93-8105-45bc-95ff-354e31340c3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":215454835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":215454835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":215454835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6809","nonce":215454835}]},"public_addr":"172.21.15.43:6801/215454835","cluster_addr":"172.21.15.43:6802/215454835","heartbeat_back_addr":"172.21.15.43:6810/215454835","heartbeat_front_addr":"172.21.15.43:6809/215454835","state":["exists","up"]},{"osd":1,"uuid":"f2c54475-fe0d-4041-b9d2-a18634c6da39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3844277216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":3844277216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":3844277216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3844277216}]},"public_addr":"172.21.15.43:6804/3844277216","cluster_addr":"172.21.15.43:6805/3844277216","heartbeat_back_addr":"172.21.15.43:6811/3844277216","heartbeat_front_addr":"172.21.15.43:6808/3844277216","state":["exists","up"]},{"osd":2,"uuid":"01b299ea-0a8e-49e4-bbc9-edacd267f29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":1388622007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":1388622007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1388622007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":1388622007}]},"public_addr":"172.21.15.164:6804/1388622007","cluster_addr":"172.21.15.164:6805/1388622007","heartbeat_back_addr":"172.21.15.164:6808/1388622007","heartbeat_front_addr":"172.21.15.164:6811/1388622007","state":["exists","up"]},{"osd":3,"uuid":"40f3ae87-d337-4a5b-a8b2-6f44aaca954e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":4179274706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":4179274706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":4179274706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":4179274706}]},"public_addr":"172.21.15.164:6806/4179274706","cluster_addr":"172.21.15.164:6807/4179274706","heartbeat_back_addr":"172.21.15.164:6813/4179274706","heartbeat_front_addr":"172.21.15.164:6814/4179274706","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T09:23:28.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-25T09:23:28.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-25T09:23:28.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-25T09:23:28.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-25T09:23:28.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.777+0000 7fa44fb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.777+0000 7fbf0e99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.777+0000 7fd6ef26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.777+0000 7f2f8cead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.787+0000 7fbf0e99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.787+0000 7fd6ef26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.789+0000 7fa44fb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.789+0000 7f2f8cead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.811 INFO:teuthology.orchestra.run.smithi043.stdout:38654705671 2024-04-25T09:23:28.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T09:23:28.814 INFO:teuthology.orchestra.run.smithi043.stdout:38654705671 2024-04-25T09:23:28.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T09:23:28.815 INFO:teuthology.orchestra.run.smithi043.stdout:38654705671 2024-04-25T09:23:28.815 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T09:23:28.818 INFO:teuthology.orchestra.run.smithi043.stdout:38654705671 2024-04-25T09:23:28.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T09:23:28.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.936+0000 7fcc7979d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.949+0000 7fcc7979d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.983+0000 7f491be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.983+0000 7ff1bd006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.994+0000 7ff1bd006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:28.994+0000 7f491be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:29.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:29.066+0000 7f3375a0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:29.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:29.076+0000 7f3375a0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:29.180 INFO:teuthology.orchestra.run.smithi043.stdout:38654705669 2024-04-25T09:23:29.191 INFO:teuthology.orchestra.run.smithi043.stdout:38654705669 2024-04-25T09:23:29.193 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705669 for osd.1 2024-04-25T09:23:29.204 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705669 for osd.2 2024-04-25T09:23:29.204 INFO:teuthology.orchestra.run.smithi043.stdout:38654705669 2024-04-25T09:23:29.218 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705669 for osd.3 2024-04-25T09:23:29.282 INFO:teuthology.orchestra.run.smithi043.stdout:38654705669 2024-04-25T09:23:29.295 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705669 for osd.0 2024-04-25T09:23:30.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T09:23:30.205 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T09:23:30.219 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T09:23:30.296 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T09:23:30.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:30.334+0000 7fefe9ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:30.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:30.348+0000 7fefe9ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:30.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:30.375+0000 7f7d5a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:30.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:30.378+0000 7f2830c17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:30.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:30.386+0000 7f7d5a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:30.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:30.388+0000 7f2830c17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:30.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:30.435+0000 7f137bcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:30.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:30.445+0000 7f137bcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:30.560 INFO:teuthology.orchestra.run.smithi043.stdout:38654705671 2024-04-25T09:23:30.574 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.1 2024-04-25T09:23:30.574 DEBUG:teuthology.parallel:result is None 2024-04-25T09:23:30.599 INFO:teuthology.orchestra.run.smithi043.stdout:38654705671 2024-04-25T09:23:30.605 INFO:teuthology.orchestra.run.smithi043.stdout:38654705671 2024-04-25T09:23:30.617 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.2 2024-04-25T09:23:30.617 DEBUG:teuthology.parallel:result is None 2024-04-25T09:23:30.618 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.3 2024-04-25T09:23:30.618 DEBUG:teuthology.parallel:result is None 2024-04-25T09:23:30.658 INFO:teuthology.orchestra.run.smithi043.stdout:38654705671 2024-04-25T09:23:30.670 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.0 2024-04-25T09:23:30.670 DEBUG:teuthology.parallel:result is None 2024-04-25T09:23:30.670 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T09:23:30.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T09:23:30.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:30.831+0000 7f435566d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:30.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:30.839+0000 7f435566d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:31.020 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:23:31.021 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T09:23:31.033 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-25T09:23:29.502559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":109808,"kb_used_data":1624,"kb_used_omap":0,"kb_used_meta":107776,"kb_avail":419320592,"statfs":{"total":429496729600,"available":429384286208,"internally_reserved":0,"allocated":1662976,"data_stored":1299808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":110362624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.922525"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.666503+0000","last_change":"2024-04-25T09:23:27.667733+0000","last_active":"2024-04-25T09:23:27.666503+0000","last_peered":"2024-04-25T09:23:27.666503+0000","last_clean":"2024-04-25T09:23:26.552349+0000","last_became_active":"2024-04-25T09:23:27.662121+0000","last_became_peered":"2024-04-25T09:23:27.662121+0000","last_unstale":"2024-04-25T09:23:27.666503+0000","last_undegraded":"2024-04-25T09:23:27.666503+0000","last_fullsized":"2024-04-25T09:23:27.666503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T09:23:28.597225+0000","last_change":"2024-04-25T09:23:27.675576+0000","last_active":"2024-04-25T09:23:28.597225+0000","last_peered":"2024-04-25T09:23:28.597225+0000","last_clean":"2024-04-25T09:23:28.597225+0000","last_became_active":"2024-04-25T09:23:27.671371+0000","last_became_peered":"2024-04-25T09:23:27.671371+0000","last_unstale":"2024-04-25T09:23:28.597225+0000","last_undegraded":"2024-04-25T09:23:28.597225+0000","last_fullsized":"2024-04-25T09:23:28.597225+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.639305+0000","last_change":"2024-04-25T09:23:27.641008+0000","last_active":"2024-04-25T09:23:27.639305+0000","last_peered":"2024-04-25T09:23:27.639305+0000","last_clean":"2024-04-25T09:23:26.552349+0000","last_became_active":"2024-04-25T09:23:27.635511+0000","last_became_peered":"2024-04-25T09:23:27.635511+0000","last_unstale":"2024-04-25T09:23:27.639305+0000","last_undegraded":"2024-04-25T09:23:27.639305+0000","last_fullsized":"2024-04-25T09:23:27.639305+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T09:23:28.593536+0000","last_change":"2024-04-25T09:23:27.631860+0000","last_active":"2024-04-25T09:23:28.593536+0000","last_peered":"2024-04-25T09:23:28.593536+0000","last_clean":"2024-04-25T09:23:28.593536+0000","last_became_active":"2024-04-25T09:23:27.627575+0000","last_became_peered":"2024-04-25T09:23:27.627575+0000","last_unstale":"2024-04-25T09:23:28.593536+0000","last_undegraded":"2024-04-25T09:23:28.593536+0000","last_fullsized":"2024-04-25T09:23:28.593536+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.640518+0000","last_change":"2024-04-25T09:23:27.642661+0000","last_active":"2024-04-25T09:23:27.640518+0000","last_peered":"2024-04-25T09:23:27.640518+0000","last_clean":"2024-04-25T09:23:26.552349+0000","last_became_active":"2024-04-25T09:23:27.635754+0000","last_became_peered":"2024-04-25T09:23:27.635754+0000","last_unstale":"2024-04-25T09:23:27.640518+0000","last_undegraded":"2024-04-25T09:23:27.640518+0000","last_fullsized":"2024-04-25T09:23:27.640518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.633496+0000","last_change":"2024-04-25T09:23:27.635085+0000","last_active":"2024-04-25T09:23:27.633496+0000","last_peered":"2024-04-25T09:23:27.633496+0000","last_clean":"2024-04-25T09:23:26.552349+0000","last_became_active":"2024-04-25T09:23:27.629956+0000","last_became_peered":"2024-04-25T09:23:27.629956+0000","last_unstale":"2024-04-25T09:23:27.633496+0000","last_undegraded":"2024-04-25T09:23:27.633496+0000","last_fullsized":"2024-04-25T09:23:27.633496+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.656144+0000","last_change":"2024-04-25T09:23:27.657105+0000","last_active":"2024-04-25T09:23:27.656144+0000","last_peered":"2024-04-25T09:23:27.656144+0000","last_clean":"2024-04-25T09:23:26.552349+0000","last_became_active":"2024-04-25T09:23:27.652630+0000","last_became_peered":"2024-04-25T09:23:27.652630+0000","last_unstale":"2024-04-25T09:23:27.656144+0000","last_undegraded":"2024-04-25T09:23:27.656144+0000","last_fullsized":"2024-04-25T09:23:27.656144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.660777+0000","last_change":"2024-04-25T09:23:27.662397+0000","last_active":"2024-04-25T09:23:27.660777+0000","last_peered":"2024-04-25T09:23:27.660777+0000","last_clean":"2024-04-25T09:23:26.552349+0000","last_became_active":"2024-04-25T09:23:27.654208+0000","last_became_peered":"2024-04-25T09:23:27.654208+0000","last_unstale":"2024-04-25T09:23:27.660777+0000","last_undegraded":"2024-04-25T09:23:27.660777+0000","last_fullsized":"2024-04-25T09:23:27.660777+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":144,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.593908+0000","last_change":"2024-04-25T09:23:24.588688+0000","last_active":"2024-04-25T09:23:27.593908+0000","last_peered":"2024-04-25T09:23:27.593908+0000","last_clean":"2024-04-25T09:23:27.593908+0000","last_became_active":"2024-04-25T09:23:24.583493+0000","last_became_peered":"2024-04-25T09:23:24.583493+0000","last_unstale":"2024-04-25T09:23:27.593908+0000","last_undegraded":"2024-04-25T09:23:27.593908+0000","last_fullsized":"2024-04-25T09:23:27.593908+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:23.538402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:23.538402+0000","last_clean_scrub_stamp":"2024-04-25T09:23:23.538402+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27836,"kb_used_data":696,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829764,"statfs":{"total":107374182400,"available":107345678336,"internally_reserved":0,"allocated":712704,"data_stored":620136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27068,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830532,"statfs":{"total":107374182400,"available":107346464768,"internally_reserved":0,"allocated":118784,"data_stored":29768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27068,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830532,"statfs":{"total":107374182400,"available":107346464768,"internally_reserved":0,"allocated":118784,"data_stored":29768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27836,"kb_used_data":696,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829764,"statfs":{"total":107374182400,"available":107345678336,"internally_reserved":0,"allocated":712704,"data_stored":620136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T09:23:31.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T09:23:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:31.155+0000 7f6ffa19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:31.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:31.163+0000 7f6ffa19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:23:31.346 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T09:23:31.358 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-25T09:23:29.502559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":109808,"kb_used_data":1624,"kb_used_omap":0,"kb_used_meta":107776,"kb_avail":419320592,"statfs":{"total":429496729600,"available":429384286208,"internally_reserved":0,"allocated":1662976,"data_stored":1299808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":110362624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.922525"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.666503+0000","last_change":"2024-04-25T09:23:27.667733+0000","last_active":"2024-04-25T09:23:27.666503+0000","last_peered":"2024-04-25T09:23:27.666503+0000","last_clean":"2024-04-25T09:23:26.552349+0000","last_became_active":"2024-04-25T09:23:27.662121+0000","last_became_peered":"2024-04-25T09:23:27.662121+0000","last_unstale":"2024-04-25T09:23:27.666503+0000","last_undegraded":"2024-04-25T09:23:27.666503+0000","last_fullsized":"2024-04-25T09:23:27.666503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T09:23:28.597225+0000","last_change":"2024-04-25T09:23:27.675576+0000","last_active":"2024-04-25T09:23:28.597225+0000","last_peered":"2024-04-25T09:23:28.597225+0000","last_clean":"2024-04-25T09:23:28.597225+0000","last_became_active":"2024-04-25T09:23:27.671371+0000","last_became_peered":"2024-04-25T09:23:27.671371+0000","last_unstale":"2024-04-25T09:23:28.597225+0000","last_undegraded":"2024-04-25T09:23:28.597225+0000","last_fullsized":"2024-04-25T09:23:28.597225+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.639305+0000","last_change":"2024-04-25T09:23:27.641008+0000","last_active":"2024-04-25T09:23:27.639305+0000","last_peered":"2024-04-25T09:23:27.639305+0000","last_clean":"2024-04-25T09:23:26.552349+0000","last_became_active":"2024-04-25T09:23:27.635511+0000","last_became_peered":"2024-04-25T09:23:27.635511+0000","last_unstale":"2024-04-25T09:23:27.639305+0000","last_undegraded":"2024-04-25T09:23:27.639305+0000","last_fullsized":"2024-04-25T09:23:27.639305+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T09:23:28.593536+0000","last_change":"2024-04-25T09:23:27.631860+0000","last_active":"2024-04-25T09:23:28.593536+0000","last_peered":"2024-04-25T09:23:28.593536+0000","last_clean":"2024-04-25T09:23:28.593536+0000","last_became_active":"2024-04-25T09:23:27.627575+0000","last_became_peered":"2024-04-25T09:23:27.627575+0000","last_unstale":"2024-04-25T09:23:28.593536+0000","last_undegraded":"2024-04-25T09:23:28.593536+0000","last_fullsized":"2024-04-25T09:23:28.593536+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.640518+0000","last_change":"2024-04-25T09:23:27.642661+0000","last_active":"2024-04-25T09:23:27.640518+0000","last_peered":"2024-04-25T09:23:27.640518+0000","last_clean":"2024-04-25T09:23:26.552349+0000","last_became_active":"2024-04-25T09:23:27.635754+0000","last_became_peered":"2024-04-25T09:23:27.635754+0000","last_unstale":"2024-04-25T09:23:27.640518+0000","last_undegraded":"2024-04-25T09:23:27.640518+0000","last_fullsized":"2024-04-25T09:23:27.640518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.633496+0000","last_change":"2024-04-25T09:23:27.635085+0000","last_active":"2024-04-25T09:23:27.633496+0000","last_peered":"2024-04-25T09:23:27.633496+0000","last_clean":"2024-04-25T09:23:26.552349+0000","last_became_active":"2024-04-25T09:23:27.629956+0000","last_became_peered":"2024-04-25T09:23:27.629956+0000","last_unstale":"2024-04-25T09:23:27.633496+0000","last_undegraded":"2024-04-25T09:23:27.633496+0000","last_fullsized":"2024-04-25T09:23:27.633496+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.656144+0000","last_change":"2024-04-25T09:23:27.657105+0000","last_active":"2024-04-25T09:23:27.656144+0000","last_peered":"2024-04-25T09:23:27.656144+0000","last_clean":"2024-04-25T09:23:26.552349+0000","last_became_active":"2024-04-25T09:23:27.652630+0000","last_became_peered":"2024-04-25T09:23:27.652630+0000","last_unstale":"2024-04-25T09:23:27.656144+0000","last_undegraded":"2024-04-25T09:23:27.656144+0000","last_fullsized":"2024-04-25T09:23:27.656144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.660777+0000","last_change":"2024-04-25T09:23:27.662397+0000","last_active":"2024-04-25T09:23:27.660777+0000","last_peered":"2024-04-25T09:23:27.660777+0000","last_clean":"2024-04-25T09:23:26.552349+0000","last_became_active":"2024-04-25T09:23:27.654208+0000","last_became_peered":"2024-04-25T09:23:27.654208+0000","last_unstale":"2024-04-25T09:23:27.660777+0000","last_undegraded":"2024-04-25T09:23:27.660777+0000","last_fullsized":"2024-04-25T09:23:27.660777+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":144,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-25T09:23:27.593908+0000","last_change":"2024-04-25T09:23:24.588688+0000","last_active":"2024-04-25T09:23:27.593908+0000","last_peered":"2024-04-25T09:23:27.593908+0000","last_clean":"2024-04-25T09:23:27.593908+0000","last_became_active":"2024-04-25T09:23:24.583493+0000","last_became_peered":"2024-04-25T09:23:24.583493+0000","last_unstale":"2024-04-25T09:23:27.593908+0000","last_undegraded":"2024-04-25T09:23:27.593908+0000","last_fullsized":"2024-04-25T09:23:27.593908+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:23.538402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:23.538402+0000","last_clean_scrub_stamp":"2024-04-25T09:23:23.538402+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27836,"kb_used_data":696,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829764,"statfs":{"total":107374182400,"available":107345678336,"internally_reserved":0,"allocated":712704,"data_stored":620136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27068,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830532,"statfs":{"total":107374182400,"available":107346464768,"internally_reserved":0,"allocated":118784,"data_stored":29768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27068,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830532,"statfs":{"total":107374182400,"available":107346464768,"internally_reserved":0,"allocated":118784,"data_stored":29768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27836,"kb_used_data":696,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829764,"statfs":{"total":107374182400,"available":107345678336,"internally_reserved":0,"allocated":712704,"data_stored":620136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T09:23:31.359 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T09:23:31.359 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T09:23:31.359 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-25T09:23:31.359 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T09:23:31.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:31.480+0000 7f87d302d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:31.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:23:31.487+0000 7f87d302d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:23:31.717 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:23:31.717 INFO:teuthology.orchestra.run.smithi043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T09:23:31.730 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-25T09:23:31.730 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T09:23:31.743 INFO:tasks.workunit:Pulling workunits from ref 472cc2b524a88306ef4535053d26658c5fd6e9db 2024-04-25T09:23:31.743 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T09:23:31.744 DEBUG:teuthology.orchestra.run.smithi043:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T09:23:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T09:23:31.759 INFO:teuthology.orchestra.run.smithi043.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-25T09:23:31.759 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-25T09:23:31.815 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-25T09:23:31.815 DEBUG:teuthology.orchestra.run.smithi043:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-25T09:23:31.871 INFO:tasks.workunit:timeout=1h 2024-04-25T09:23:31.871 INFO:tasks.workunit:cleanup=True 2024-04-25T09:23:31.871 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 472cc2b524a88306ef4535053d26658c5fd6e9db 2024-04-25T09:23:31.927 INFO:tasks.workunit.client.0.smithi043.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T09:24:25.728 INFO:tasks.workunit.client.0.smithi043.stderr:Note: switching to '472cc2b524a88306ef4535053d26658c5fd6e9db'. 2024-04-25T09:24:25.728 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-04-25T09:24:25.728 INFO:tasks.workunit.client.0.smithi043.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T09:24:25.728 INFO:tasks.workunit.client.0.smithi043.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T09:24:25.728 INFO:tasks.workunit.client.0.smithi043.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T09:24:25.729 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-04-25T09:24:25.729 INFO:tasks.workunit.client.0.smithi043.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T09:24:25.729 INFO:tasks.workunit.client.0.smithi043.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T09:24:25.729 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-04-25T09:24:25.730 INFO:tasks.workunit.client.0.smithi043.stderr: git switch -c 2024-04-25T09:24:25.730 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-04-25T09:24:25.730 INFO:tasks.workunit.client.0.smithi043.stderr:Or undo this operation with: 2024-04-25T09:24:25.730 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-04-25T09:24:25.730 INFO:tasks.workunit.client.0.smithi043.stderr: git switch - 2024-04-25T09:24:25.730 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-04-25T09:24:25.730 INFO:tasks.workunit.client.0.smithi043.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T09:24:25.730 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-04-25T09:24:25.730 INFO:tasks.workunit.client.0.smithi043.stderr:HEAD is now at 472cc2b524a Merge pull request #56942 from Svelar/fix_tox_cephadm 2024-04-25T09:24:25.731 DEBUG:teuthology.orchestra.run.smithi043:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T09:24:25.851 INFO:tasks.workunit.client.0.smithi043.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T09:24:25.853 INFO:tasks.workunit.client.0.smithi043.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T09:24:25.853 INFO:tasks.workunit.client.0.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T09:24:26.499 INFO:tasks.workunit.client.0.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T09:24:26.558 INFO:tasks.workunit.client.0.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T09:24:26.600 INFO:tasks.workunit.client.0.smithi043.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T09:24:26.602 INFO:tasks.workunit.client.0.smithi043.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T09:24:26.602 INFO:tasks.workunit.client.0.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T09:24:26.642 INFO:tasks.workunit.client.0.smithi043.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T09:24:26.645 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T09:24:26.646 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T09:24:26.662 INFO:tasks.workunit:Running workunits matching rados/test_python.sh on client.0... 2024-04-25T09:24:26.663 INFO:tasks.workunit:Running workunit rados/test_python.sh... 2024-04-25T09:24:26.663 DEBUG:teuthology.orchestra.run.smithi043:workunit test rados/test_python.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=472cc2b524a88306ef4535053d26658c5fd6e9db TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh -m 'not (wait or tier or ec or bench or stats)' 2024-04-25T09:24:26.721 INFO:tasks.workunit.client.0.smithi043.stderr:+ ceph osd pool create rbd 2024-04-25T09:24:26.814 INFO:tasks.workunit.client.0.smithi043.stderr:2024-04-25T09:24:26.822+0000 7fd81706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:24:26.819 INFO:tasks.workunit.client.0.smithi043.stderr:2024-04-25T09:24:26.833+0000 7fd81706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:24:27.127 INFO:tasks.workunit.client.0.smithi043.stderr:pool 'rbd' already exists 2024-04-25T09:24:27.139 INFO:tasks.workunit.client.0.smithi043.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh 2024-04-25T09:24:27.139 INFO:tasks.workunit.client.0.smithi043.stderr:+ python3 -m pytest -v /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/../../../src/test/pybind/test_rados.py -m 'not (wait or tier or ec or bench or stats)' 2024-04-25T09:24:27.315 INFO:tasks.workunit.client.0.smithi043.stdout:============================= test session starts ============================== 2024-04-25T09:24:27.315 INFO:tasks.workunit.client.0.smithi043.stdout:platform linux -- Python 3.9.18, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3 2024-04-25T09:24:27.316 INFO:tasks.workunit.client.0.smithi043.stdout:cachedir: .pytest_cache 2024-04-25T09:24:27.316 INFO:tasks.workunit.client.0.smithi043.stdout:rootdir: /home/ubuntu/cephtest/clone.client.0/src/test/pybind, configfile: pytest.ini 2024-04-25T09:24:27.601 INFO:tasks.workunit.client.0.smithi043.stdout:collecting ... collected 90 items / 7 deselected / 83 selected 2024-04-25T09:24:27.601 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-04-25T09:24:27.613 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::test_rados_init_error PASSED [ 1%] 2024-04-25T09:24:27.710 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::test_rados_init PASSED [ 2%] 2024-04-25T09:24:27.734 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::test_ioctx_context_manager PASSED [ 3%] 2024-04-25T09:24:27.748 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::test_parse_argv PASSED [ 4%] 2024-04-25T09:24:27.755 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::test_parse_argv_empty_str PASSED [ 6%] 2024-04-25T09:24:27.770 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestRadosStateError::test_configuring PASSED [ 7%] 2024-04-25T09:24:27.795 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestRadosStateError::test_connected PASSED [ 8%] 2024-04-25T09:24:27.821 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestRadosStateError::test_shutdown PASSED [ 9%] 2024-04-25T09:24:27.852 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestRados::test_ping_monitor PASSED [ 10%] 2024-04-25T09:24:27.880 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestRados::test_annotations PASSED [ 12%] 2024-04-25T09:24:29.333 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestRados::test_create PASSED [ 13%] 2024-04-25T09:24:31.338 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestRados::test_create_utf8 PASSED [ 14%] 2024-04-25T09:24:33.349 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestRados::test_pool_lookup_utf8 PASSED [ 15%] 2024-04-25T09:24:35.356 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestRados::test_eexist PASSED [ 16%] 2024-04-25T09:24:43.392 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestRados::test_list_pools PASSED [ 18%] 2024-04-25T09:24:43.433 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestRados::test_get_fsid PASSED [ 19%] 2024-04-25T09:24:44.396 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestRados::test_blocklist_add PASSED [ 20%] 2024-04-25T09:24:44.433 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestRados::test_monitor_log PASSED [ 21%] 2024-04-25T09:24:47.319 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_get_last_version PASSED [ 22%] 2024-04-25T09:24:50.329 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_get_stats PASSED [ 24%] 2024-04-25T09:24:54.345 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_write PASSED [ 25%] 2024-04-25T09:24:58.359 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_write_full PASSED [ 26%] 2024-04-25T09:25:02.380 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_writesame PASSED [ 27%] 2024-04-25T09:25:06.397 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_append PASSED [ 28%] 2024-04-25T09:25:10.407 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_write_zeros PASSED [ 30%] 2024-04-25T09:25:14.430 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_trunc PASSED [ 31%] 2024-04-25T09:25:18.444 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_cmpext PASSED [ 32%] 2024-04-25T09:25:22.463 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_list_objects_empty PASSED [ 33%] 2024-04-25T09:25:26.474 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_list_objects PASSED [ 34%] 2024-04-25T09:25:30.323 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_list_ns_objects PASSED [ 36%] 2024-04-25T09:25:34.333 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_xattrs PASSED [ 37%] 2024-04-25T09:25:38.349 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_obj_xattrs PASSED [ 38%] 2024-04-25T09:25:41.325 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_get_pool_id PASSED [ 39%] 2024-04-25T09:25:44.332 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_get_pool_name PASSED [ 40%] 2024-04-25T09:25:49.358 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_create_snap PASSED [ 42%] 2024-04-25T09:25:52.329 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_list_snaps_empty PASSED [ 43%] 2024-04-25T09:25:58.351 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_list_snaps PASSED [ 44%] 2024-04-25T09:26:02.334 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_lookup_snap PASSED [ 45%] 2024-04-25T09:26:06.346 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_snap_timestamp PASSED [ 46%] 2024-04-25T09:26:11.333 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_remove_snap PASSED [ 48%] 2024-04-25T09:26:17.357 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_snap_rollback PASSED [ 49%] 2024-04-25T09:26:23.379 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_snap_rollback_removed PASSED [ 50%] 2024-04-25T09:26:29.404 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_snap_read PASSED [ 51%] 2024-04-25T09:26:33.425 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_set_omap PASSED [ 53%] 2024-04-25T09:26:37.437 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_set_omap_aio PASSED [ 54%] 2024-04-25T09:26:41.340 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_write_ops PASSED [ 55%] 2024-04-25T09:26:45.350 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_execute_op PASSED [ 56%] 2024-04-25T09:26:49.367 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_writesame_op PASSED [ 57%] 2024-04-25T09:26:53.389 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_get_omap_vals_by_keys PASSED [ 59%] 2024-04-25T09:26:57.404 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_get_omap_keys PASSED [ 60%] 2024-04-25T09:27:01.344 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_clear_omap PASSED [ 61%] 2024-04-25T09:27:05.426 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_remove_omap_range2 PASSED [ 62%] 2024-04-25T09:27:09.443 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_omap_cmp PASSED [ 63%] 2024-04-25T09:27:13.457 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_cmpext_op PASSED [ 65%] 2024-04-25T09:27:17.471 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_xattrs_op PASSED [ 66%] 2024-04-25T09:27:20.347 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_locator PASSED [ 67%] 2024-04-25T09:27:24.492 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_operate_aio_write_op PASSED [ 68%] 2024-04-25T09:27:28.503 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_aio_write PASSED [ 69%] 2024-04-25T09:27:32.352 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_aio_cmpext PASSED [ 71%] 2024-04-25T09:27:36.369 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_aio_rmxattr PASSED [ 72%] 2024-04-25T09:27:40.353 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_aio_write_no_comp_ref PASSED [ 73%] 2024-04-25T09:27:44.398 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_aio_append PASSED [ 74%] 2024-04-25T09:27:48.415 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_aio_write_full PASSED [ 75%] 2024-04-25T09:27:52.432 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_aio_writesame PASSED [ 77%] 2024-04-25T09:27:56.448 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_aio_stat PASSED [ 78%] 2024-04-25T09:28:00.357 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_aio_remove PASSED [ 79%] 2024-04-25T09:28:04.370 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_lock PASSED [ 80%] 2024-04-25T09:28:08.381 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_execute PASSED [ 81%] 2024-04-25T09:28:12.402 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_aio_execute PASSED [ 83%] 2024-04-25T09:28:16.417 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_aio_setxattr PASSED [ 84%] 2024-04-25T09:28:25.374 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_applications PASSED [ 85%] 2024-04-25T09:28:28.382 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_service_daemon PASSED [ 86%] 2024-04-25T09:28:31.365 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx::test_alignment PASSED [ 87%] 2024-04-25T09:28:34.377 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx2::test_get_last_version PASSED [ 89%] 2024-04-25T09:28:37.388 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoctx2::test_get_stats PASSED [ 90%] 2024-04-25T09:28:40.404 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestObject::test_read PASSED [ 91%] 2024-04-25T09:28:43.418 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestObject::test_seek PASSED [ 92%] 2024-04-25T09:28:46.430 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestObject::test_write PASSED [ 93%] 2024-04-25T09:28:54.454 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestIoCtxSelfManagedSnaps::test PASSED [ 95%] 2024-04-25T09:28:54.485 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestCommand::test_monmap_dump PASSED [ 96%] 2024-04-25T09:28:55.370 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestCommand::test_ceph_osd_pool_create_utf8 PASSED [ 97%] 2024-04-25T09:29:01.467 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestWatchNotify::test PASSED [ 98%] 2024-04-25T09:29:07.488 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py::TestWatchNotify::test_aio_notify PASSED [100%] 2024-04-25T09:29:07.488 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-04-25T09:29:07.488 INFO:tasks.workunit.client.0.smithi043.stdout:=============================== warnings summary =============================== 2024-04-25T09:29:07.488 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py:210 2024-04-25T09:29:07.488 INFO:tasks.workunit.client.0.smithi043.stdout: /home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py:210: DeprecationWarning: invalid escape sequence \- 2024-04-25T09:29:07.488 INFO:tasks.workunit.client.0.smithi043.stdout: assert re.match('[0-9a-f\-]{36}', fsid, re.I) 2024-04-25T09:29:07.488 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-04-25T09:29:07.488 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py:955 2024-04-25T09:29:07.488 INFO:tasks.workunit.client.0.smithi043.stdout: /home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py:955: PytestUnknownMarkWarning: Unknown pytest.mark.wait - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html 2024-04-25T09:29:07.489 INFO:tasks.workunit.client.0.smithi043.stdout: @pytest.mark.wait 2024-04-25T09:29:07.489 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-04-25T09:29:07.489 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py:991 2024-04-25T09:29:07.489 INFO:tasks.workunit.client.0.smithi043.stdout: /home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py:991: PytestUnknownMarkWarning: Unknown pytest.mark.wait - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html 2024-04-25T09:29:07.489 INFO:tasks.workunit.client.0.smithi043.stdout: @pytest.mark.wait 2024-04-25T09:29:07.489 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-04-25T09:29:07.489 INFO:tasks.workunit.client.0.smithi043.stdout:../../../clone.client.0/src/test/pybind/test_rados.py:1019 2024-04-25T09:29:07.489 INFO:tasks.workunit.client.0.smithi043.stdout: /home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py:1019: PytestUnknownMarkWarning: Unknown pytest.mark.wait - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html 2024-04-25T09:29:07.489 INFO:tasks.workunit.client.0.smithi043.stdout: @pytest.mark.wait 2024-04-25T09:29:07.489 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-04-25T09:29:07.489 INFO:tasks.workunit.client.0.smithi043.stdout::210 2024-04-25T09:29:07.490 INFO:tasks.workunit.client.0.smithi043.stdout::210 2024-04-25T09:29:07.490 INFO:tasks.workunit.client.0.smithi043.stdout::210 2024-04-25T09:29:07.490 INFO:tasks.workunit.client.0.smithi043.stdout::210 2024-04-25T09:29:07.490 INFO:tasks.workunit.client.0.smithi043.stdout::210 2024-04-25T09:29:07.490 INFO:tasks.workunit.client.0.smithi043.stdout::210 2024-04-25T09:29:07.490 INFO:tasks.workunit.client.0.smithi043.stdout::210 2024-04-25T09:29:07.490 INFO:tasks.workunit.client.0.smithi043.stdout::210 2024-04-25T09:29:07.490 INFO:tasks.workunit.client.0.smithi043.stdout::210 2024-04-25T09:29:07.490 INFO:tasks.workunit.client.0.smithi043.stdout: :210: DeprecationWarning: invalid escape sequence \- 2024-04-25T09:29:07.490 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-04-25T09:29:07.490 INFO:tasks.workunit.client.0.smithi043.stdout:-- Docs: https://docs.pytest.org/en/stable/warnings.html 2024-04-25T09:29:07.490 INFO:tasks.workunit.client.0.smithi043.stdout:========== 83 passed, 7 deselected, 13 warnings in 280.17s (0:04:40) =========== 2024-04-25T09:29:07.515 INFO:tasks.workunit.client.0.smithi043.stderr:+ exit 0 2024-04-25T09:29:07.515 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:29:07.516 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:29:07.584 INFO:tasks.workunit:Stopping ["rados/test_python.sh -m 'not (wait or tier or ec or bench or stats)'"] on client.0... 2024-04-25T09:29:07.584 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T09:29:07.943 DEBUG:teuthology.parallel:result is None 2024-04-25T09:29:07.944 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:29:07.970 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:29:07.970 DEBUG:teuthology.orchestra.run.smithi043:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-25T09:29:08.033 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:29:08.033 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T09:29:08.044 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T09:29:08.044 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T09:29:08.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:08.195+0000 7fc316e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:08.204+0000 7fc316e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:08.385 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:29:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T09:29:08.397 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-04-25T09:29:07.574919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":150112,"kb_used_data":4864,"kb_used_omap":0,"kb_used_meta":145216,"kb_avail":419280288,"statfs":{"total":429496729600,"available":429343014912,"internally_reserved":0,"allocated":4980736,"data_stored":2402095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":148701184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.091018"},"pg_stats":[{"pgid":"72.f","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.641387+0000","last_change":"2024-04-25T09:28:56.604081+0000","last_active":"2024-04-25T09:29:04.641387+0000","last_peered":"2024-04-25T09:29:04.641387+0000","last_clean":"2024-04-25T09:29:04.641387+0000","last_became_active":"2024-04-25T09:28:56.594281+0000","last_became_peered":"2024-04-25T09:28:56.594281+0000","last_unstale":"2024-04-25T09:29:04.641387+0000","last_undegraded":"2024-04-25T09:29:04.641387+0000","last_fullsized":"2024-04-25T09:29:04.641387+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.e","version":"0'0","reported_seq":23,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.583228+0000","last_change":"2024-04-25T09:28:56.585836+0000","last_active":"2024-04-25T09:29:04.583228+0000","last_peered":"2024-04-25T09:29:04.583228+0000","last_clean":"2024-04-25T09:29:04.583228+0000","last_became_active":"2024-04-25T09:28:56.581636+0000","last_became_peered":"2024-04-25T09:28:56.581636+0000","last_unstale":"2024-04-25T09:29:04.583228+0000","last_undegraded":"2024-04-25T09:29:04.583228+0000","last_fullsized":"2024-04-25T09:29:04.583228+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.d","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.634887+0000","last_change":"2024-04-25T09:28:56.578739+0000","last_active":"2024-04-25T09:29:04.634887+0000","last_peered":"2024-04-25T09:29:04.634887+0000","last_clean":"2024-04-25T09:29:04.634887+0000","last_became_active":"2024-04-25T09:28:56.569011+0000","last_became_peered":"2024-04-25T09:28:56.569011+0000","last_unstale":"2024-04-25T09:29:04.634887+0000","last_undegraded":"2024-04-25T09:29:04.634887+0000","last_fullsized":"2024-04-25T09:29:04.634887+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.c","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.585835+0000","last_change":"2024-04-25T09:28:56.582420+0000","last_active":"2024-04-25T09:29:04.585835+0000","last_peered":"2024-04-25T09:29:04.585835+0000","last_clean":"2024-04-25T09:29:04.585835+0000","last_became_active":"2024-04-25T09:28:56.566867+0000","last_became_peered":"2024-04-25T09:28:56.566867+0000","last_unstale":"2024-04-25T09:29:04.585835+0000","last_undegraded":"2024-04-25T09:29:04.585835+0000","last_fullsized":"2024-04-25T09:29:04.585835+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.b","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.626898+0000","last_change":"2024-04-25T09:28:56.632525+0000","last_active":"2024-04-25T09:29:04.626898+0000","last_peered":"2024-04-25T09:29:04.626898+0000","last_clean":"2024-04-25T09:29:04.626898+0000","last_became_active":"2024-04-25T09:28:56.625254+0000","last_became_peered":"2024-04-25T09:28:56.625254+0000","last_unstale":"2024-04-25T09:29:04.626898+0000","last_undegraded":"2024-04-25T09:29:04.626898+0000","last_fullsized":"2024-04-25T09:29:04.626898+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.a","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.614115+0000","last_change":"2024-04-25T09:28:56.570786+0000","last_active":"2024-04-25T09:29:04.614115+0000","last_peered":"2024-04-25T09:29:04.614115+0000","last_clean":"2024-04-25T09:29:04.614115+0000","last_became_active":"2024-04-25T09:28:56.556905+0000","last_became_peered":"2024-04-25T09:28:56.556905+0000","last_unstale":"2024-04-25T09:29:04.614115+0000","last_undegraded":"2024-04-25T09:29:04.614115+0000","last_fullsized":"2024-04-25T09:29:04.614115+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.9","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.617091+0000","last_change":"2024-04-25T09:28:56.588495+0000","last_active":"2024-04-25T09:29:04.617091+0000","last_peered":"2024-04-25T09:29:04.617091+0000","last_clean":"2024-04-25T09:29:04.617091+0000","last_became_active":"2024-04-25T09:28:56.574219+0000","last_became_peered":"2024-04-25T09:28:56.574219+0000","last_unstale":"2024-04-25T09:29:04.617091+0000","last_undegraded":"2024-04-25T09:29:04.617091+0000","last_fullsized":"2024-04-25T09:29:04.617091+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.8","version":"0'0","reported_seq":23,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.617968+0000","last_change":"2024-04-25T09:28:56.628274+0000","last_active":"2024-04-25T09:29:04.617968+0000","last_peered":"2024-04-25T09:29:04.617968+0000","last_clean":"2024-04-25T09:29:04.617968+0000","last_became_active":"2024-04-25T09:28:56.620437+0000","last_became_peered":"2024-04-25T09:28:56.620437+0000","last_unstale":"2024-04-25T09:29:04.617968+0000","last_undegraded":"2024-04-25T09:29:04.617968+0000","last_fullsized":"2024-04-25T09:29:04.617968+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.7","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.615542+0000","last_change":"2024-04-25T09:28:56.541105+0000","last_active":"2024-04-25T09:29:04.615542+0000","last_peered":"2024-04-25T09:29:04.615542+0000","last_clean":"2024-04-25T09:29:04.615542+0000","last_became_active":"2024-04-25T09:28:56.532134+0000","last_became_peered":"2024-04-25T09:28:56.532134+0000","last_unstale":"2024-04-25T09:29:04.615542+0000","last_undegraded":"2024-04-25T09:29:04.615542+0000","last_fullsized":"2024-04-25T09:29:04.615542+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.6","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.636709+0000","last_change":"2024-04-25T09:28:56.601065+0000","last_active":"2024-04-25T09:29:04.636709+0000","last_peered":"2024-04-25T09:29:04.636709+0000","last_clean":"2024-04-25T09:29:04.636709+0000","last_became_active":"2024-04-25T09:28:56.584773+0000","last_became_peered":"2024-04-25T09:28:56.584773+0000","last_unstale":"2024-04-25T09:29:04.636709+0000","last_undegraded":"2024-04-25T09:29:04.636709+0000","last_fullsized":"2024-04-25T09:29:04.636709+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.5","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.608000+0000","last_change":"2024-04-25T09:28:56.589439+0000","last_active":"2024-04-25T09:29:04.608000+0000","last_peered":"2024-04-25T09:29:04.608000+0000","last_clean":"2024-04-25T09:29:04.608000+0000","last_became_active":"2024-04-25T09:28:56.579671+0000","last_became_peered":"2024-04-25T09:28:56.579671+0000","last_unstale":"2024-04-25T09:29:04.608000+0000","last_undegraded":"2024-04-25T09:29:04.608000+0000","last_fullsized":"2024-04-25T09:29:04.608000+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.4","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.551457+0000","last_change":"2024-04-25T09:28:56.555285+0000","last_active":"2024-04-25T09:29:04.551457+0000","last_peered":"2024-04-25T09:29:04.551457+0000","last_clean":"2024-04-25T09:29:04.551457+0000","last_became_active":"2024-04-25T09:28:56.546535+0000","last_became_peered":"2024-04-25T09:28:56.546535+0000","last_unstale":"2024-04-25T09:29:04.551457+0000","last_undegraded":"2024-04-25T09:29:04.551457+0000","last_fullsized":"2024-04-25T09:29:04.551457+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.3","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.610411+0000","last_change":"2024-04-25T09:28:56.555708+0000","last_active":"2024-04-25T09:29:04.610411+0000","last_peered":"2024-04-25T09:29:04.610411+0000","last_clean":"2024-04-25T09:29:04.610411+0000","last_became_active":"2024-04-25T09:28:56.547591+0000","last_became_peered":"2024-04-25T09:28:56.547591+0000","last_unstale":"2024-04-25T09:29:04.610411+0000","last_undegraded":"2024-04-25T09:29:04.610411+0000","last_fullsized":"2024-04-25T09:29:04.610411+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.2","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.533005+0000","last_change":"2024-04-25T09:28:56.538188+0000","last_active":"2024-04-25T09:29:04.533005+0000","last_peered":"2024-04-25T09:29:04.533005+0000","last_clean":"2024-04-25T09:29:04.533005+0000","last_became_active":"2024-04-25T09:28:56.531302+0000","last_became_peered":"2024-04-25T09:28:56.531302+0000","last_unstale":"2024-04-25T09:29:04.533005+0000","last_undegraded":"2024-04-25T09:29:04.533005+0000","last_fullsized":"2024-04-25T09:29:04.533005+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.1","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.504442+0000","last_change":"2024-04-25T09:28:56.560898+0000","last_active":"2024-04-25T09:29:04.504442+0000","last_peered":"2024-04-25T09:29:04.504442+0000","last_clean":"2024-04-25T09:29:04.504442+0000","last_became_active":"2024-04-25T09:28:56.551023+0000","last_became_peered":"2024-04-25T09:28:56.551023+0000","last_unstale":"2024-04-25T09:29:04.504442+0000","last_undegraded":"2024-04-25T09:29:04.504442+0000","last_fullsized":"2024-04-25T09:29:04.504442+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.0","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.611781+0000","last_change":"2024-04-25T09:28:56.576712+0000","last_active":"2024-04-25T09:29:04.611781+0000","last_peered":"2024-04-25T09:29:04.611781+0000","last_clean":"2024-04-25T09:29:04.611781+0000","last_became_active":"2024-04-25T09:28:56.561446+0000","last_became_peered":"2024-04-25T09:28:56.561446+0000","last_unstale":"2024-04-25T09:29:04.611781+0000","last_undegraded":"2024-04-25T09:29:04.611781+0000","last_fullsized":"2024-04-25T09:29:04.611781+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":843,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.610832+0000","last_change":"2024-04-25T09:23:24.588688+0000","last_active":"2024-04-25T09:29:04.610832+0000","last_peered":"2024-04-25T09:29:04.610832+0000","last_clean":"2024-04-25T09:29:04.610832+0000","last_became_active":"2024-04-25T09:23:24.583493+0000","last_became_peered":"2024-04-25T09:23:24.583493+0000","last_unstale":"2024-04-25T09:29:04.610832+0000","last_undegraded":"2024-04-25T09:29:04.610832+0000","last_fullsized":"2024-04-25T09:29:04.610832+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:23.538402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:23.538402+0000","last_clean_scrub_stamp":"2024-04-25T09:23:23.538402+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.558713+0000","last_change":"2024-04-25T09:23:27.662397+0000","last_active":"2024-04-25T09:29:04.558713+0000","last_peered":"2024-04-25T09:29:04.558713+0000","last_clean":"2024-04-25T09:29:04.558713+0000","last_became_active":"2024-04-25T09:23:27.654208+0000","last_became_peered":"2024-04-25T09:23:27.654208+0000","last_unstale":"2024-04-25T09:29:04.558713+0000","last_undegraded":"2024-04-25T09:29:04.558713+0000","last_fullsized":"2024-04-25T09:29:04.558713+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":705,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.633547+0000","last_change":"2024-04-25T09:23:27.657105+0000","last_active":"2024-04-25T09:29:04.633547+0000","last_peered":"2024-04-25T09:29:04.633547+0000","last_clean":"2024-04-25T09:29:04.633547+0000","last_became_active":"2024-04-25T09:23:27.652630+0000","last_became_peered":"2024-04-25T09:23:27.652630+0000","last_unstale":"2024-04-25T09:29:04.633547+0000","last_undegraded":"2024-04-25T09:29:04.633547+0000","last_fullsized":"2024-04-25T09:29:04.633547+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.544980+0000","last_change":"2024-04-25T09:23:27.635085+0000","last_active":"2024-04-25T09:29:04.544980+0000","last_peered":"2024-04-25T09:29:04.544980+0000","last_clean":"2024-04-25T09:29:04.544980+0000","last_became_active":"2024-04-25T09:23:27.629956+0000","last_became_peered":"2024-04-25T09:23:27.629956+0000","last_unstale":"2024-04-25T09:29:04.544980+0000","last_undegraded":"2024-04-25T09:29:04.544980+0000","last_fullsized":"2024-04-25T09:29:04.544980+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":705,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.615186+0000","last_change":"2024-04-25T09:23:27.642661+0000","last_active":"2024-04-25T09:29:04.615186+0000","last_peered":"2024-04-25T09:29:04.615186+0000","last_clean":"2024-04-25T09:29:04.615186+0000","last_became_active":"2024-04-25T09:23:27.635754+0000","last_became_peered":"2024-04-25T09:23:27.635754+0000","last_unstale":"2024-04-25T09:29:04.615186+0000","last_undegraded":"2024-04-25T09:29:04.615186+0000","last_fullsized":"2024-04-25T09:29:04.615186+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":697,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.605955+0000","last_change":"2024-04-25T09:23:27.631860+0000","last_active":"2024-04-25T09:29:04.605955+0000","last_peered":"2024-04-25T09:29:04.605955+0000","last_clean":"2024-04-25T09:29:04.605955+0000","last_became_active":"2024-04-25T09:23:27.627575+0000","last_became_peered":"2024-04-25T09:23:27.627575+0000","last_unstale":"2024-04-25T09:29:04.605955+0000","last_undegraded":"2024-04-25T09:29:04.605955+0000","last_fullsized":"2024-04-25T09:29:04.605955+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":707,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.655721+0000","last_change":"2024-04-25T09:23:27.641008+0000","last_active":"2024-04-25T09:29:04.655721+0000","last_peered":"2024-04-25T09:29:04.655721+0000","last_clean":"2024-04-25T09:29:04.655721+0000","last_became_active":"2024-04-25T09:23:27.635511+0000","last_became_peered":"2024-04-25T09:23:27.635511+0000","last_unstale":"2024-04-25T09:29:04.655721+0000","last_undegraded":"2024-04-25T09:29:04.655721+0000","last_fullsized":"2024-04-25T09:29:04.655721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":697,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.613085+0000","last_change":"2024-04-25T09:23:27.675576+0000","last_active":"2024-04-25T09:29:04.613085+0000","last_peered":"2024-04-25T09:29:04.613085+0000","last_clean":"2024-04-25T09:29:04.613085+0000","last_became_active":"2024-04-25T09:23:27.671371+0000","last_became_peered":"2024-04-25T09:23:27.671371+0000","last_unstale":"2024-04-25T09:29:04.613085+0000","last_undegraded":"2024-04-25T09:29:04.613085+0000","last_fullsized":"2024-04-25T09:29:04.613085+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.528911+0000","last_change":"2024-04-25T09:23:27.667733+0000","last_active":"2024-04-25T09:29:04.528911+0000","last_peered":"2024-04-25T09:29:04.528911+0000","last_clean":"2024-04-25T09:29:04.528911+0000","last_became_active":"2024-04-25T09:23:27.662121+0000","last_became_peered":"2024-04-25T09:23:27.662121+0000","last_unstale":"2024-04-25T09:29:04.528911+0000","last_undegraded":"2024-04-25T09:29:04.528911+0000","last_fullsized":"2024-04-25T09:29:04.528911+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":72,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36780,"kb_used_data":1508,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104820820,"statfs":{"total":107374182400,"available":107336519680,"internally_reserved":0,"allocated":1544192,"data_stored":895709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36196,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104821404,"statfs":{"total":107374182400,"available":107337117696,"internally_reserved":0,"allocated":946176,"data_stored":305338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36196,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104821404,"statfs":{"total":107374182400,"available":107337117696,"internally_reserved":0,"allocated":946176,"data_stored":305338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":40940,"kb_used_data":1508,"kb_used_omap":0,"kb_used_meta":39424,"kb_avail":104816660,"statfs":{"total":107374182400,"available":107332259840,"internally_reserved":0,"allocated":1544192,"data_stored":895710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40370176},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T09:29:08.399 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T09:29:08.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:08.522+0000 7f9f2b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:08.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:08.531+0000 7f9f2b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:08.798 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:29:08.799 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T09:29:08.815 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-04-25T09:29:07.574919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":150112,"kb_used_data":4864,"kb_used_omap":0,"kb_used_meta":145216,"kb_avail":419280288,"statfs":{"total":429496729600,"available":429343014912,"internally_reserved":0,"allocated":4980736,"data_stored":2402095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":148701184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.091018"},"pg_stats":[{"pgid":"72.f","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.641387+0000","last_change":"2024-04-25T09:28:56.604081+0000","last_active":"2024-04-25T09:29:04.641387+0000","last_peered":"2024-04-25T09:29:04.641387+0000","last_clean":"2024-04-25T09:29:04.641387+0000","last_became_active":"2024-04-25T09:28:56.594281+0000","last_became_peered":"2024-04-25T09:28:56.594281+0000","last_unstale":"2024-04-25T09:29:04.641387+0000","last_undegraded":"2024-04-25T09:29:04.641387+0000","last_fullsized":"2024-04-25T09:29:04.641387+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.e","version":"0'0","reported_seq":23,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.583228+0000","last_change":"2024-04-25T09:28:56.585836+0000","last_active":"2024-04-25T09:29:04.583228+0000","last_peered":"2024-04-25T09:29:04.583228+0000","last_clean":"2024-04-25T09:29:04.583228+0000","last_became_active":"2024-04-25T09:28:56.581636+0000","last_became_peered":"2024-04-25T09:28:56.581636+0000","last_unstale":"2024-04-25T09:29:04.583228+0000","last_undegraded":"2024-04-25T09:29:04.583228+0000","last_fullsized":"2024-04-25T09:29:04.583228+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.d","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.634887+0000","last_change":"2024-04-25T09:28:56.578739+0000","last_active":"2024-04-25T09:29:04.634887+0000","last_peered":"2024-04-25T09:29:04.634887+0000","last_clean":"2024-04-25T09:29:04.634887+0000","last_became_active":"2024-04-25T09:28:56.569011+0000","last_became_peered":"2024-04-25T09:28:56.569011+0000","last_unstale":"2024-04-25T09:29:04.634887+0000","last_undegraded":"2024-04-25T09:29:04.634887+0000","last_fullsized":"2024-04-25T09:29:04.634887+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.c","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.585835+0000","last_change":"2024-04-25T09:28:56.582420+0000","last_active":"2024-04-25T09:29:04.585835+0000","last_peered":"2024-04-25T09:29:04.585835+0000","last_clean":"2024-04-25T09:29:04.585835+0000","last_became_active":"2024-04-25T09:28:56.566867+0000","last_became_peered":"2024-04-25T09:28:56.566867+0000","last_unstale":"2024-04-25T09:29:04.585835+0000","last_undegraded":"2024-04-25T09:29:04.585835+0000","last_fullsized":"2024-04-25T09:29:04.585835+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.b","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.626898+0000","last_change":"2024-04-25T09:28:56.632525+0000","last_active":"2024-04-25T09:29:04.626898+0000","last_peered":"2024-04-25T09:29:04.626898+0000","last_clean":"2024-04-25T09:29:04.626898+0000","last_became_active":"2024-04-25T09:28:56.625254+0000","last_became_peered":"2024-04-25T09:28:56.625254+0000","last_unstale":"2024-04-25T09:29:04.626898+0000","last_undegraded":"2024-04-25T09:29:04.626898+0000","last_fullsized":"2024-04-25T09:29:04.626898+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.a","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.614115+0000","last_change":"2024-04-25T09:28:56.570786+0000","last_active":"2024-04-25T09:29:04.614115+0000","last_peered":"2024-04-25T09:29:04.614115+0000","last_clean":"2024-04-25T09:29:04.614115+0000","last_became_active":"2024-04-25T09:28:56.556905+0000","last_became_peered":"2024-04-25T09:28:56.556905+0000","last_unstale":"2024-04-25T09:29:04.614115+0000","last_undegraded":"2024-04-25T09:29:04.614115+0000","last_fullsized":"2024-04-25T09:29:04.614115+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.9","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.617091+0000","last_change":"2024-04-25T09:28:56.588495+0000","last_active":"2024-04-25T09:29:04.617091+0000","last_peered":"2024-04-25T09:29:04.617091+0000","last_clean":"2024-04-25T09:29:04.617091+0000","last_became_active":"2024-04-25T09:28:56.574219+0000","last_became_peered":"2024-04-25T09:28:56.574219+0000","last_unstale":"2024-04-25T09:29:04.617091+0000","last_undegraded":"2024-04-25T09:29:04.617091+0000","last_fullsized":"2024-04-25T09:29:04.617091+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.8","version":"0'0","reported_seq":23,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.617968+0000","last_change":"2024-04-25T09:28:56.628274+0000","last_active":"2024-04-25T09:29:04.617968+0000","last_peered":"2024-04-25T09:29:04.617968+0000","last_clean":"2024-04-25T09:29:04.617968+0000","last_became_active":"2024-04-25T09:28:56.620437+0000","last_became_peered":"2024-04-25T09:28:56.620437+0000","last_unstale":"2024-04-25T09:29:04.617968+0000","last_undegraded":"2024-04-25T09:29:04.617968+0000","last_fullsized":"2024-04-25T09:29:04.617968+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.7","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.615542+0000","last_change":"2024-04-25T09:28:56.541105+0000","last_active":"2024-04-25T09:29:04.615542+0000","last_peered":"2024-04-25T09:29:04.615542+0000","last_clean":"2024-04-25T09:29:04.615542+0000","last_became_active":"2024-04-25T09:28:56.532134+0000","last_became_peered":"2024-04-25T09:28:56.532134+0000","last_unstale":"2024-04-25T09:29:04.615542+0000","last_undegraded":"2024-04-25T09:29:04.615542+0000","last_fullsized":"2024-04-25T09:29:04.615542+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.6","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.636709+0000","last_change":"2024-04-25T09:28:56.601065+0000","last_active":"2024-04-25T09:29:04.636709+0000","last_peered":"2024-04-25T09:29:04.636709+0000","last_clean":"2024-04-25T09:29:04.636709+0000","last_became_active":"2024-04-25T09:28:56.584773+0000","last_became_peered":"2024-04-25T09:28:56.584773+0000","last_unstale":"2024-04-25T09:29:04.636709+0000","last_undegraded":"2024-04-25T09:29:04.636709+0000","last_fullsized":"2024-04-25T09:29:04.636709+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.5","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.608000+0000","last_change":"2024-04-25T09:28:56.589439+0000","last_active":"2024-04-25T09:29:04.608000+0000","last_peered":"2024-04-25T09:29:04.608000+0000","last_clean":"2024-04-25T09:29:04.608000+0000","last_became_active":"2024-04-25T09:28:56.579671+0000","last_became_peered":"2024-04-25T09:28:56.579671+0000","last_unstale":"2024-04-25T09:29:04.608000+0000","last_undegraded":"2024-04-25T09:29:04.608000+0000","last_fullsized":"2024-04-25T09:29:04.608000+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.4","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.551457+0000","last_change":"2024-04-25T09:28:56.555285+0000","last_active":"2024-04-25T09:29:04.551457+0000","last_peered":"2024-04-25T09:29:04.551457+0000","last_clean":"2024-04-25T09:29:04.551457+0000","last_became_active":"2024-04-25T09:28:56.546535+0000","last_became_peered":"2024-04-25T09:28:56.546535+0000","last_unstale":"2024-04-25T09:29:04.551457+0000","last_undegraded":"2024-04-25T09:29:04.551457+0000","last_fullsized":"2024-04-25T09:29:04.551457+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.3","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.610411+0000","last_change":"2024-04-25T09:28:56.555708+0000","last_active":"2024-04-25T09:29:04.610411+0000","last_peered":"2024-04-25T09:29:04.610411+0000","last_clean":"2024-04-25T09:29:04.610411+0000","last_became_active":"2024-04-25T09:28:56.547591+0000","last_became_peered":"2024-04-25T09:28:56.547591+0000","last_unstale":"2024-04-25T09:29:04.610411+0000","last_undegraded":"2024-04-25T09:29:04.610411+0000","last_fullsized":"2024-04-25T09:29:04.610411+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.2","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.533005+0000","last_change":"2024-04-25T09:28:56.538188+0000","last_active":"2024-04-25T09:29:04.533005+0000","last_peered":"2024-04-25T09:29:04.533005+0000","last_clean":"2024-04-25T09:29:04.533005+0000","last_became_active":"2024-04-25T09:28:56.531302+0000","last_became_peered":"2024-04-25T09:28:56.531302+0000","last_unstale":"2024-04-25T09:29:04.533005+0000","last_undegraded":"2024-04-25T09:29:04.533005+0000","last_fullsized":"2024-04-25T09:29:04.533005+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.1","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.504442+0000","last_change":"2024-04-25T09:28:56.560898+0000","last_active":"2024-04-25T09:29:04.504442+0000","last_peered":"2024-04-25T09:29:04.504442+0000","last_clean":"2024-04-25T09:29:04.504442+0000","last_became_active":"2024-04-25T09:28:56.551023+0000","last_became_peered":"2024-04-25T09:28:56.551023+0000","last_unstale":"2024-04-25T09:29:04.504442+0000","last_undegraded":"2024-04-25T09:29:04.504442+0000","last_fullsized":"2024-04-25T09:29:04.504442+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.0","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.611781+0000","last_change":"2024-04-25T09:28:56.576712+0000","last_active":"2024-04-25T09:29:04.611781+0000","last_peered":"2024-04-25T09:29:04.611781+0000","last_clean":"2024-04-25T09:29:04.611781+0000","last_became_active":"2024-04-25T09:28:56.561446+0000","last_became_peered":"2024-04-25T09:28:56.561446+0000","last_unstale":"2024-04-25T09:29:04.611781+0000","last_undegraded":"2024-04-25T09:29:04.611781+0000","last_fullsized":"2024-04-25T09:29:04.611781+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":843,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.610832+0000","last_change":"2024-04-25T09:23:24.588688+0000","last_active":"2024-04-25T09:29:04.610832+0000","last_peered":"2024-04-25T09:29:04.610832+0000","last_clean":"2024-04-25T09:29:04.610832+0000","last_became_active":"2024-04-25T09:23:24.583493+0000","last_became_peered":"2024-04-25T09:23:24.583493+0000","last_unstale":"2024-04-25T09:29:04.610832+0000","last_undegraded":"2024-04-25T09:29:04.610832+0000","last_fullsized":"2024-04-25T09:29:04.610832+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:23.538402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:23.538402+0000","last_clean_scrub_stamp":"2024-04-25T09:23:23.538402+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.558713+0000","last_change":"2024-04-25T09:23:27.662397+0000","last_active":"2024-04-25T09:29:04.558713+0000","last_peered":"2024-04-25T09:29:04.558713+0000","last_clean":"2024-04-25T09:29:04.558713+0000","last_became_active":"2024-04-25T09:23:27.654208+0000","last_became_peered":"2024-04-25T09:23:27.654208+0000","last_unstale":"2024-04-25T09:29:04.558713+0000","last_undegraded":"2024-04-25T09:29:04.558713+0000","last_fullsized":"2024-04-25T09:29:04.558713+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":705,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.633547+0000","last_change":"2024-04-25T09:23:27.657105+0000","last_active":"2024-04-25T09:29:04.633547+0000","last_peered":"2024-04-25T09:29:04.633547+0000","last_clean":"2024-04-25T09:29:04.633547+0000","last_became_active":"2024-04-25T09:23:27.652630+0000","last_became_peered":"2024-04-25T09:23:27.652630+0000","last_unstale":"2024-04-25T09:29:04.633547+0000","last_undegraded":"2024-04-25T09:29:04.633547+0000","last_fullsized":"2024-04-25T09:29:04.633547+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.544980+0000","last_change":"2024-04-25T09:23:27.635085+0000","last_active":"2024-04-25T09:29:04.544980+0000","last_peered":"2024-04-25T09:29:04.544980+0000","last_clean":"2024-04-25T09:29:04.544980+0000","last_became_active":"2024-04-25T09:23:27.629956+0000","last_became_peered":"2024-04-25T09:23:27.629956+0000","last_unstale":"2024-04-25T09:29:04.544980+0000","last_undegraded":"2024-04-25T09:29:04.544980+0000","last_fullsized":"2024-04-25T09:29:04.544980+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":705,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.615186+0000","last_change":"2024-04-25T09:23:27.642661+0000","last_active":"2024-04-25T09:29:04.615186+0000","last_peered":"2024-04-25T09:29:04.615186+0000","last_clean":"2024-04-25T09:29:04.615186+0000","last_became_active":"2024-04-25T09:23:27.635754+0000","last_became_peered":"2024-04-25T09:23:27.635754+0000","last_unstale":"2024-04-25T09:29:04.615186+0000","last_undegraded":"2024-04-25T09:29:04.615186+0000","last_fullsized":"2024-04-25T09:29:04.615186+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":697,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.605955+0000","last_change":"2024-04-25T09:23:27.631860+0000","last_active":"2024-04-25T09:29:04.605955+0000","last_peered":"2024-04-25T09:29:04.605955+0000","last_clean":"2024-04-25T09:29:04.605955+0000","last_became_active":"2024-04-25T09:23:27.627575+0000","last_became_peered":"2024-04-25T09:23:27.627575+0000","last_unstale":"2024-04-25T09:29:04.605955+0000","last_undegraded":"2024-04-25T09:29:04.605955+0000","last_fullsized":"2024-04-25T09:29:04.605955+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":707,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.655721+0000","last_change":"2024-04-25T09:23:27.641008+0000","last_active":"2024-04-25T09:29:04.655721+0000","last_peered":"2024-04-25T09:29:04.655721+0000","last_clean":"2024-04-25T09:29:04.655721+0000","last_became_active":"2024-04-25T09:23:27.635511+0000","last_became_peered":"2024-04-25T09:23:27.635511+0000","last_unstale":"2024-04-25T09:29:04.655721+0000","last_undegraded":"2024-04-25T09:29:04.655721+0000","last_fullsized":"2024-04-25T09:29:04.655721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":697,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.613085+0000","last_change":"2024-04-25T09:23:27.675576+0000","last_active":"2024-04-25T09:29:04.613085+0000","last_peered":"2024-04-25T09:29:04.613085+0000","last_clean":"2024-04-25T09:29:04.613085+0000","last_became_active":"2024-04-25T09:23:27.671371+0000","last_became_peered":"2024-04-25T09:23:27.671371+0000","last_unstale":"2024-04-25T09:29:04.613085+0000","last_undegraded":"2024-04-25T09:29:04.613085+0000","last_fullsized":"2024-04-25T09:29:04.613085+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.528911+0000","last_change":"2024-04-25T09:23:27.667733+0000","last_active":"2024-04-25T09:29:04.528911+0000","last_peered":"2024-04-25T09:29:04.528911+0000","last_clean":"2024-04-25T09:29:04.528911+0000","last_became_active":"2024-04-25T09:23:27.662121+0000","last_became_peered":"2024-04-25T09:23:27.662121+0000","last_unstale":"2024-04-25T09:29:04.528911+0000","last_undegraded":"2024-04-25T09:29:04.528911+0000","last_fullsized":"2024-04-25T09:29:04.528911+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":72,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36780,"kb_used_data":1508,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104820820,"statfs":{"total":107374182400,"available":107336519680,"internally_reserved":0,"allocated":1544192,"data_stored":895709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36196,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104821404,"statfs":{"total":107374182400,"available":107337117696,"internally_reserved":0,"allocated":946176,"data_stored":305338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36196,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104821404,"statfs":{"total":107374182400,"available":107337117696,"internally_reserved":0,"allocated":946176,"data_stored":305338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":40940,"kb_used_data":1508,"kb_used_omap":0,"kb_used_meta":39424,"kb_avail":104816660,"statfs":{"total":107374182400,"available":107332259840,"internally_reserved":0,"allocated":1544192,"data_stored":895710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40370176},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T09:29:08.816 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T09:29:08.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T09:29:08.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:08.965+0000 7f7793e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:08.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:08.973+0000 7f7793e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:09.152 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:29:09.153 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T09:29:09.165 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-04-25T09:29:07.574919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":150112,"kb_used_data":4864,"kb_used_omap":0,"kb_used_meta":145216,"kb_avail":419280288,"statfs":{"total":429496729600,"available":429343014912,"internally_reserved":0,"allocated":4980736,"data_stored":2402095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":148701184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.091018"},"pg_stats":[{"pgid":"72.f","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.641387+0000","last_change":"2024-04-25T09:28:56.604081+0000","last_active":"2024-04-25T09:29:04.641387+0000","last_peered":"2024-04-25T09:29:04.641387+0000","last_clean":"2024-04-25T09:29:04.641387+0000","last_became_active":"2024-04-25T09:28:56.594281+0000","last_became_peered":"2024-04-25T09:28:56.594281+0000","last_unstale":"2024-04-25T09:29:04.641387+0000","last_undegraded":"2024-04-25T09:29:04.641387+0000","last_fullsized":"2024-04-25T09:29:04.641387+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.e","version":"0'0","reported_seq":23,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.583228+0000","last_change":"2024-04-25T09:28:56.585836+0000","last_active":"2024-04-25T09:29:04.583228+0000","last_peered":"2024-04-25T09:29:04.583228+0000","last_clean":"2024-04-25T09:29:04.583228+0000","last_became_active":"2024-04-25T09:28:56.581636+0000","last_became_peered":"2024-04-25T09:28:56.581636+0000","last_unstale":"2024-04-25T09:29:04.583228+0000","last_undegraded":"2024-04-25T09:29:04.583228+0000","last_fullsized":"2024-04-25T09:29:04.583228+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.d","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.634887+0000","last_change":"2024-04-25T09:28:56.578739+0000","last_active":"2024-04-25T09:29:04.634887+0000","last_peered":"2024-04-25T09:29:04.634887+0000","last_clean":"2024-04-25T09:29:04.634887+0000","last_became_active":"2024-04-25T09:28:56.569011+0000","last_became_peered":"2024-04-25T09:28:56.569011+0000","last_unstale":"2024-04-25T09:29:04.634887+0000","last_undegraded":"2024-04-25T09:29:04.634887+0000","last_fullsized":"2024-04-25T09:29:04.634887+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.c","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.585835+0000","last_change":"2024-04-25T09:28:56.582420+0000","last_active":"2024-04-25T09:29:04.585835+0000","last_peered":"2024-04-25T09:29:04.585835+0000","last_clean":"2024-04-25T09:29:04.585835+0000","last_became_active":"2024-04-25T09:28:56.566867+0000","last_became_peered":"2024-04-25T09:28:56.566867+0000","last_unstale":"2024-04-25T09:29:04.585835+0000","last_undegraded":"2024-04-25T09:29:04.585835+0000","last_fullsized":"2024-04-25T09:29:04.585835+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.b","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.626898+0000","last_change":"2024-04-25T09:28:56.632525+0000","last_active":"2024-04-25T09:29:04.626898+0000","last_peered":"2024-04-25T09:29:04.626898+0000","last_clean":"2024-04-25T09:29:04.626898+0000","last_became_active":"2024-04-25T09:28:56.625254+0000","last_became_peered":"2024-04-25T09:28:56.625254+0000","last_unstale":"2024-04-25T09:29:04.626898+0000","last_undegraded":"2024-04-25T09:29:04.626898+0000","last_fullsized":"2024-04-25T09:29:04.626898+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.a","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.614115+0000","last_change":"2024-04-25T09:28:56.570786+0000","last_active":"2024-04-25T09:29:04.614115+0000","last_peered":"2024-04-25T09:29:04.614115+0000","last_clean":"2024-04-25T09:29:04.614115+0000","last_became_active":"2024-04-25T09:28:56.556905+0000","last_became_peered":"2024-04-25T09:28:56.556905+0000","last_unstale":"2024-04-25T09:29:04.614115+0000","last_undegraded":"2024-04-25T09:29:04.614115+0000","last_fullsized":"2024-04-25T09:29:04.614115+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.9","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.617091+0000","last_change":"2024-04-25T09:28:56.588495+0000","last_active":"2024-04-25T09:29:04.617091+0000","last_peered":"2024-04-25T09:29:04.617091+0000","last_clean":"2024-04-25T09:29:04.617091+0000","last_became_active":"2024-04-25T09:28:56.574219+0000","last_became_peered":"2024-04-25T09:28:56.574219+0000","last_unstale":"2024-04-25T09:29:04.617091+0000","last_undegraded":"2024-04-25T09:29:04.617091+0000","last_fullsized":"2024-04-25T09:29:04.617091+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.8","version":"0'0","reported_seq":23,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.617968+0000","last_change":"2024-04-25T09:28:56.628274+0000","last_active":"2024-04-25T09:29:04.617968+0000","last_peered":"2024-04-25T09:29:04.617968+0000","last_clean":"2024-04-25T09:29:04.617968+0000","last_became_active":"2024-04-25T09:28:56.620437+0000","last_became_peered":"2024-04-25T09:28:56.620437+0000","last_unstale":"2024-04-25T09:29:04.617968+0000","last_undegraded":"2024-04-25T09:29:04.617968+0000","last_fullsized":"2024-04-25T09:29:04.617968+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.7","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.615542+0000","last_change":"2024-04-25T09:28:56.541105+0000","last_active":"2024-04-25T09:29:04.615542+0000","last_peered":"2024-04-25T09:29:04.615542+0000","last_clean":"2024-04-25T09:29:04.615542+0000","last_became_active":"2024-04-25T09:28:56.532134+0000","last_became_peered":"2024-04-25T09:28:56.532134+0000","last_unstale":"2024-04-25T09:29:04.615542+0000","last_undegraded":"2024-04-25T09:29:04.615542+0000","last_fullsized":"2024-04-25T09:29:04.615542+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.6","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.636709+0000","last_change":"2024-04-25T09:28:56.601065+0000","last_active":"2024-04-25T09:29:04.636709+0000","last_peered":"2024-04-25T09:29:04.636709+0000","last_clean":"2024-04-25T09:29:04.636709+0000","last_became_active":"2024-04-25T09:28:56.584773+0000","last_became_peered":"2024-04-25T09:28:56.584773+0000","last_unstale":"2024-04-25T09:29:04.636709+0000","last_undegraded":"2024-04-25T09:29:04.636709+0000","last_fullsized":"2024-04-25T09:29:04.636709+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.5","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.608000+0000","last_change":"2024-04-25T09:28:56.589439+0000","last_active":"2024-04-25T09:29:04.608000+0000","last_peered":"2024-04-25T09:29:04.608000+0000","last_clean":"2024-04-25T09:29:04.608000+0000","last_became_active":"2024-04-25T09:28:56.579671+0000","last_became_peered":"2024-04-25T09:28:56.579671+0000","last_unstale":"2024-04-25T09:29:04.608000+0000","last_undegraded":"2024-04-25T09:29:04.608000+0000","last_fullsized":"2024-04-25T09:29:04.608000+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.4","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.551457+0000","last_change":"2024-04-25T09:28:56.555285+0000","last_active":"2024-04-25T09:29:04.551457+0000","last_peered":"2024-04-25T09:29:04.551457+0000","last_clean":"2024-04-25T09:29:04.551457+0000","last_became_active":"2024-04-25T09:28:56.546535+0000","last_became_peered":"2024-04-25T09:28:56.546535+0000","last_unstale":"2024-04-25T09:29:04.551457+0000","last_undegraded":"2024-04-25T09:29:04.551457+0000","last_fullsized":"2024-04-25T09:29:04.551457+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.3","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.610411+0000","last_change":"2024-04-25T09:28:56.555708+0000","last_active":"2024-04-25T09:29:04.610411+0000","last_peered":"2024-04-25T09:29:04.610411+0000","last_clean":"2024-04-25T09:29:04.610411+0000","last_became_active":"2024-04-25T09:28:56.547591+0000","last_became_peered":"2024-04-25T09:28:56.547591+0000","last_unstale":"2024-04-25T09:29:04.610411+0000","last_undegraded":"2024-04-25T09:29:04.610411+0000","last_fullsized":"2024-04-25T09:29:04.610411+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.2","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.533005+0000","last_change":"2024-04-25T09:28:56.538188+0000","last_active":"2024-04-25T09:29:04.533005+0000","last_peered":"2024-04-25T09:29:04.533005+0000","last_clean":"2024-04-25T09:29:04.533005+0000","last_became_active":"2024-04-25T09:28:56.531302+0000","last_became_peered":"2024-04-25T09:28:56.531302+0000","last_unstale":"2024-04-25T09:29:04.533005+0000","last_undegraded":"2024-04-25T09:29:04.533005+0000","last_fullsized":"2024-04-25T09:29:04.533005+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.1","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.504442+0000","last_change":"2024-04-25T09:28:56.560898+0000","last_active":"2024-04-25T09:29:04.504442+0000","last_peered":"2024-04-25T09:29:04.504442+0000","last_clean":"2024-04-25T09:29:04.504442+0000","last_became_active":"2024-04-25T09:28:56.551023+0000","last_became_peered":"2024-04-25T09:28:56.551023+0000","last_unstale":"2024-04-25T09:29:04.504442+0000","last_undegraded":"2024-04-25T09:29:04.504442+0000","last_fullsized":"2024-04-25T09:29:04.504442+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.0","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.611781+0000","last_change":"2024-04-25T09:28:56.576712+0000","last_active":"2024-04-25T09:29:04.611781+0000","last_peered":"2024-04-25T09:29:04.611781+0000","last_clean":"2024-04-25T09:29:04.611781+0000","last_became_active":"2024-04-25T09:28:56.561446+0000","last_became_peered":"2024-04-25T09:28:56.561446+0000","last_unstale":"2024-04-25T09:29:04.611781+0000","last_undegraded":"2024-04-25T09:29:04.611781+0000","last_fullsized":"2024-04-25T09:29:04.611781+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":843,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.610832+0000","last_change":"2024-04-25T09:23:24.588688+0000","last_active":"2024-04-25T09:29:04.610832+0000","last_peered":"2024-04-25T09:29:04.610832+0000","last_clean":"2024-04-25T09:29:04.610832+0000","last_became_active":"2024-04-25T09:23:24.583493+0000","last_became_peered":"2024-04-25T09:23:24.583493+0000","last_unstale":"2024-04-25T09:29:04.610832+0000","last_undegraded":"2024-04-25T09:29:04.610832+0000","last_fullsized":"2024-04-25T09:29:04.610832+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:23.538402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:23.538402+0000","last_clean_scrub_stamp":"2024-04-25T09:23:23.538402+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.558713+0000","last_change":"2024-04-25T09:23:27.662397+0000","last_active":"2024-04-25T09:29:04.558713+0000","last_peered":"2024-04-25T09:29:04.558713+0000","last_clean":"2024-04-25T09:29:04.558713+0000","last_became_active":"2024-04-25T09:23:27.654208+0000","last_became_peered":"2024-04-25T09:23:27.654208+0000","last_unstale":"2024-04-25T09:29:04.558713+0000","last_undegraded":"2024-04-25T09:29:04.558713+0000","last_fullsized":"2024-04-25T09:29:04.558713+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":705,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.633547+0000","last_change":"2024-04-25T09:23:27.657105+0000","last_active":"2024-04-25T09:29:04.633547+0000","last_peered":"2024-04-25T09:29:04.633547+0000","last_clean":"2024-04-25T09:29:04.633547+0000","last_became_active":"2024-04-25T09:23:27.652630+0000","last_became_peered":"2024-04-25T09:23:27.652630+0000","last_unstale":"2024-04-25T09:29:04.633547+0000","last_undegraded":"2024-04-25T09:29:04.633547+0000","last_fullsized":"2024-04-25T09:29:04.633547+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.544980+0000","last_change":"2024-04-25T09:23:27.635085+0000","last_active":"2024-04-25T09:29:04.544980+0000","last_peered":"2024-04-25T09:29:04.544980+0000","last_clean":"2024-04-25T09:29:04.544980+0000","last_became_active":"2024-04-25T09:23:27.629956+0000","last_became_peered":"2024-04-25T09:23:27.629956+0000","last_unstale":"2024-04-25T09:29:04.544980+0000","last_undegraded":"2024-04-25T09:29:04.544980+0000","last_fullsized":"2024-04-25T09:29:04.544980+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":705,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.615186+0000","last_change":"2024-04-25T09:23:27.642661+0000","last_active":"2024-04-25T09:29:04.615186+0000","last_peered":"2024-04-25T09:29:04.615186+0000","last_clean":"2024-04-25T09:29:04.615186+0000","last_became_active":"2024-04-25T09:23:27.635754+0000","last_became_peered":"2024-04-25T09:23:27.635754+0000","last_unstale":"2024-04-25T09:29:04.615186+0000","last_undegraded":"2024-04-25T09:29:04.615186+0000","last_fullsized":"2024-04-25T09:29:04.615186+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":697,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.605955+0000","last_change":"2024-04-25T09:23:27.631860+0000","last_active":"2024-04-25T09:29:04.605955+0000","last_peered":"2024-04-25T09:29:04.605955+0000","last_clean":"2024-04-25T09:29:04.605955+0000","last_became_active":"2024-04-25T09:23:27.627575+0000","last_became_peered":"2024-04-25T09:23:27.627575+0000","last_unstale":"2024-04-25T09:29:04.605955+0000","last_undegraded":"2024-04-25T09:29:04.605955+0000","last_fullsized":"2024-04-25T09:29:04.605955+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":707,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.655721+0000","last_change":"2024-04-25T09:23:27.641008+0000","last_active":"2024-04-25T09:29:04.655721+0000","last_peered":"2024-04-25T09:29:04.655721+0000","last_clean":"2024-04-25T09:29:04.655721+0000","last_became_active":"2024-04-25T09:23:27.635511+0000","last_became_peered":"2024-04-25T09:23:27.635511+0000","last_unstale":"2024-04-25T09:29:04.655721+0000","last_undegraded":"2024-04-25T09:29:04.655721+0000","last_fullsized":"2024-04-25T09:29:04.655721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":697,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.613085+0000","last_change":"2024-04-25T09:23:27.675576+0000","last_active":"2024-04-25T09:29:04.613085+0000","last_peered":"2024-04-25T09:29:04.613085+0000","last_clean":"2024-04-25T09:29:04.613085+0000","last_became_active":"2024-04-25T09:23:27.671371+0000","last_became_peered":"2024-04-25T09:23:27.671371+0000","last_unstale":"2024-04-25T09:29:04.613085+0000","last_undegraded":"2024-04-25T09:29:04.613085+0000","last_fullsized":"2024-04-25T09:29:04.613085+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.528911+0000","last_change":"2024-04-25T09:23:27.667733+0000","last_active":"2024-04-25T09:29:04.528911+0000","last_peered":"2024-04-25T09:29:04.528911+0000","last_clean":"2024-04-25T09:29:04.528911+0000","last_became_active":"2024-04-25T09:23:27.662121+0000","last_became_peered":"2024-04-25T09:23:27.662121+0000","last_unstale":"2024-04-25T09:29:04.528911+0000","last_undegraded":"2024-04-25T09:29:04.528911+0000","last_fullsized":"2024-04-25T09:29:04.528911+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":72,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36780,"kb_used_data":1508,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104820820,"statfs":{"total":107374182400,"available":107336519680,"internally_reserved":0,"allocated":1544192,"data_stored":895709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36196,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104821404,"statfs":{"total":107374182400,"available":107337117696,"internally_reserved":0,"allocated":946176,"data_stored":305338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36196,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104821404,"statfs":{"total":107374182400,"available":107337117696,"internally_reserved":0,"allocated":946176,"data_stored":305338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":40940,"kb_used_data":1508,"kb_used_omap":0,"kb_used_meta":39424,"kb_avail":104816660,"statfs":{"total":107374182400,"available":107332259840,"internally_reserved":0,"allocated":1544192,"data_stored":895710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40370176},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T09:29:09.166 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T09:29:09.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:09.288+0000 7fd2da664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:09.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:09.297+0000 7fd2da664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:09.480 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:29:09.481 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":292,"fsid":"827f1403-0dbf-4d12-8151-d66aeea7580e","created":"2024-04-25T09:23:15.321458+0000","modified":"2024-04-25T09:29:07.495051+0000","last_up_change":"2024-04-25T09:23:21.536091+0000","last_in_change":"2024-04-25T09:23:17.464337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":74,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T09:23:23.521590+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T09:23:25.673624+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":72,"pool_name":"黅","create_time":"2024-04-25T09:28:54.529638+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"285","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2376fa93-8105-45bc-95ff-354e31340c3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":289,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":215454835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":215454835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":215454835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6809","nonce":215454835}]},"public_addr":"172.21.15.43:6801/215454835","cluster_addr":"172.21.15.43:6802/215454835","heartbeat_back_addr":"172.21.15.43:6810/215454835","heartbeat_front_addr":"172.21.15.43:6809/215454835","state":["exists","up"]},{"osd":1,"uuid":"f2c54475-fe0d-4041-b9d2-a18634c6da39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":289,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3844277216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":3844277216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":3844277216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3844277216}]},"public_addr":"172.21.15.43:6804/3844277216","cluster_addr":"172.21.15.43:6805/3844277216","heartbeat_back_addr":"172.21.15.43:6811/3844277216","heartbeat_front_addr":"172.21.15.43:6808/3844277216","state":["exists","up"]},{"osd":2,"uuid":"01b299ea-0a8e-49e4-bbc9-edacd267f29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":289,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":1388622007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":1388622007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1388622007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":1388622007}]},"public_addr":"172.21.15.164:6804/1388622007","cluster_addr":"172.21.15.164:6805/1388622007","heartbeat_back_addr":"172.21.15.164:6808/1388622007","heartbeat_front_addr":"172.21.15.164:6811/1388622007","state":["exists","up"]},{"osd":3,"uuid":"40f3ae87-d337-4a5b-a8b2-6f44aaca954e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":289,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":4179274706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":4179274706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":4179274706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":4179274706}]},"public_addr":"172.21.15.164:6806/4179274706","cluster_addr":"172.21.15.164:6807/4179274706","heartbeat_back_addr":"172.21.15.164:6813/4179274706","heartbeat_front_addr":"172.21.15.164:6814/4179274706","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":27,"snaps":[{"begin":1,"length":1}]},{"pool":32,"snaps":[{"begin":1,"length":1}]},{"pool":33,"snaps":[{"begin":1,"length":1}]},{"pool":34,"snaps":[{"begin":1,"length":1}]},{"pool":35,"snaps":[{"begin":1,"length":1}]},{"pool":71,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T09:29:10.495 INFO:tasks.ceph:Scrubbing osd.0 2024-04-25T09:29:10.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-25T09:29:10.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:10.620+0000 7f42612ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:10.632+0000 7f42612ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:10.631 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T09:29:10.631 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "" 2024-04-25T09:29:10.631 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T09:29:10.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-25T09:29:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:10.765+0000 7ff11d2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:10.774+0000 7ff11d2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:10.962 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 0 to deep-scrub 2024-04-25T09:29:10.984 INFO:tasks.ceph:Scrubbing osd.1 2024-04-25T09:29:10.984 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-25T09:29:11.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:11.113+0000 7fd3d755d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:11.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:11.126+0000 7fd3d755d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:11.125 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T09:29:11.125 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "" 2024-04-25T09:29:11.125 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T09:29:11.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-25T09:29:11.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:11.259+0000 7f2eecaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:11.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:11.269+0000 7f2eecaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:11.451 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 1 to deep-scrub 2024-04-25T09:29:11.468 INFO:tasks.ceph:Scrubbing osd.2 2024-04-25T09:29:11.468 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-25T09:29:11.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:11.591+0000 7f41d1f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:11.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:11.604+0000 7f41d1f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:11.605 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T09:29:11.606 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "" 2024-04-25T09:29:11.606 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T09:29:11.617 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-25T09:29:11.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:11.743+0000 7f07b9817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:11.752+0000 7f07b9817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:11.947 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 2 to deep-scrub 2024-04-25T09:29:11.963 INFO:tasks.ceph:Scrubbing osd.3 2024-04-25T09:29:11.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-25T09:29:12.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:12.085+0000 7fefe439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:12.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:12.098+0000 7fefe439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:12.097 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T09:29:12.097 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "" 2024-04-25T09:29:12.098 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T09:29:12.108 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-25T09:29:12.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:12.231+0000 7f0921f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:12.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:12.241+0000 7f0921f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:12.428 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 3 to deep-scrub 2024-04-25T09:29:12.443 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T09:29:12.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:12.568+0000 7ff704c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:12.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:12.579+0000 7ff704c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:12.761 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:29:12.761 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T09:29:12.773 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-04-25T09:29:11.575853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":150112,"kb_used_data":4864,"kb_used_omap":0,"kb_used_meta":145216,"kb_avail":419280288,"statfs":{"total":429496729600,"available":429343014912,"internally_reserved":0,"allocated":4980736,"data_stored":2402095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":148701184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.084510"},"pg_stats":[{"pgid":"72.f","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.641387+0000","last_change":"2024-04-25T09:28:56.604081+0000","last_active":"2024-04-25T09:29:04.641387+0000","last_peered":"2024-04-25T09:29:04.641387+0000","last_clean":"2024-04-25T09:29:04.641387+0000","last_became_active":"2024-04-25T09:28:56.594281+0000","last_became_peered":"2024-04-25T09:28:56.594281+0000","last_unstale":"2024-04-25T09:29:04.641387+0000","last_undegraded":"2024-04-25T09:29:04.641387+0000","last_fullsized":"2024-04-25T09:29:04.641387+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.e","version":"0'0","reported_seq":23,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.583228+0000","last_change":"2024-04-25T09:28:56.585836+0000","last_active":"2024-04-25T09:29:04.583228+0000","last_peered":"2024-04-25T09:29:04.583228+0000","last_clean":"2024-04-25T09:29:04.583228+0000","last_became_active":"2024-04-25T09:28:56.581636+0000","last_became_peered":"2024-04-25T09:28:56.581636+0000","last_unstale":"2024-04-25T09:29:04.583228+0000","last_undegraded":"2024-04-25T09:29:04.583228+0000","last_fullsized":"2024-04-25T09:29:04.583228+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.d","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.634887+0000","last_change":"2024-04-25T09:28:56.578739+0000","last_active":"2024-04-25T09:29:04.634887+0000","last_peered":"2024-04-25T09:29:04.634887+0000","last_clean":"2024-04-25T09:29:04.634887+0000","last_became_active":"2024-04-25T09:28:56.569011+0000","last_became_peered":"2024-04-25T09:28:56.569011+0000","last_unstale":"2024-04-25T09:29:04.634887+0000","last_undegraded":"2024-04-25T09:29:04.634887+0000","last_fullsized":"2024-04-25T09:29:04.634887+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.c","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.585835+0000","last_change":"2024-04-25T09:28:56.582420+0000","last_active":"2024-04-25T09:29:04.585835+0000","last_peered":"2024-04-25T09:29:04.585835+0000","last_clean":"2024-04-25T09:29:04.585835+0000","last_became_active":"2024-04-25T09:28:56.566867+0000","last_became_peered":"2024-04-25T09:28:56.566867+0000","last_unstale":"2024-04-25T09:29:04.585835+0000","last_undegraded":"2024-04-25T09:29:04.585835+0000","last_fullsized":"2024-04-25T09:29:04.585835+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.b","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.626898+0000","last_change":"2024-04-25T09:28:56.632525+0000","last_active":"2024-04-25T09:29:04.626898+0000","last_peered":"2024-04-25T09:29:04.626898+0000","last_clean":"2024-04-25T09:29:04.626898+0000","last_became_active":"2024-04-25T09:28:56.625254+0000","last_became_peered":"2024-04-25T09:28:56.625254+0000","last_unstale":"2024-04-25T09:29:04.626898+0000","last_undegraded":"2024-04-25T09:29:04.626898+0000","last_fullsized":"2024-04-25T09:29:04.626898+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.a","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.614115+0000","last_change":"2024-04-25T09:28:56.570786+0000","last_active":"2024-04-25T09:29:04.614115+0000","last_peered":"2024-04-25T09:29:04.614115+0000","last_clean":"2024-04-25T09:29:04.614115+0000","last_became_active":"2024-04-25T09:28:56.556905+0000","last_became_peered":"2024-04-25T09:28:56.556905+0000","last_unstale":"2024-04-25T09:29:04.614115+0000","last_undegraded":"2024-04-25T09:29:04.614115+0000","last_fullsized":"2024-04-25T09:29:04.614115+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.9","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.617091+0000","last_change":"2024-04-25T09:28:56.588495+0000","last_active":"2024-04-25T09:29:04.617091+0000","last_peered":"2024-04-25T09:29:04.617091+0000","last_clean":"2024-04-25T09:29:04.617091+0000","last_became_active":"2024-04-25T09:28:56.574219+0000","last_became_peered":"2024-04-25T09:28:56.574219+0000","last_unstale":"2024-04-25T09:29:04.617091+0000","last_undegraded":"2024-04-25T09:29:04.617091+0000","last_fullsized":"2024-04-25T09:29:04.617091+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.8","version":"0'0","reported_seq":23,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.617968+0000","last_change":"2024-04-25T09:28:56.628274+0000","last_active":"2024-04-25T09:29:04.617968+0000","last_peered":"2024-04-25T09:29:04.617968+0000","last_clean":"2024-04-25T09:29:04.617968+0000","last_became_active":"2024-04-25T09:28:56.620437+0000","last_became_peered":"2024-04-25T09:28:56.620437+0000","last_unstale":"2024-04-25T09:29:04.617968+0000","last_undegraded":"2024-04-25T09:29:04.617968+0000","last_fullsized":"2024-04-25T09:29:04.617968+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.7","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.615542+0000","last_change":"2024-04-25T09:28:56.541105+0000","last_active":"2024-04-25T09:29:04.615542+0000","last_peered":"2024-04-25T09:29:04.615542+0000","last_clean":"2024-04-25T09:29:04.615542+0000","last_became_active":"2024-04-25T09:28:56.532134+0000","last_became_peered":"2024-04-25T09:28:56.532134+0000","last_unstale":"2024-04-25T09:29:04.615542+0000","last_undegraded":"2024-04-25T09:29:04.615542+0000","last_fullsized":"2024-04-25T09:29:04.615542+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.6","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.636709+0000","last_change":"2024-04-25T09:28:56.601065+0000","last_active":"2024-04-25T09:29:04.636709+0000","last_peered":"2024-04-25T09:29:04.636709+0000","last_clean":"2024-04-25T09:29:04.636709+0000","last_became_active":"2024-04-25T09:28:56.584773+0000","last_became_peered":"2024-04-25T09:28:56.584773+0000","last_unstale":"2024-04-25T09:29:04.636709+0000","last_undegraded":"2024-04-25T09:29:04.636709+0000","last_fullsized":"2024-04-25T09:29:04.636709+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.5","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.608000+0000","last_change":"2024-04-25T09:28:56.589439+0000","last_active":"2024-04-25T09:29:04.608000+0000","last_peered":"2024-04-25T09:29:04.608000+0000","last_clean":"2024-04-25T09:29:04.608000+0000","last_became_active":"2024-04-25T09:28:56.579671+0000","last_became_peered":"2024-04-25T09:28:56.579671+0000","last_unstale":"2024-04-25T09:29:04.608000+0000","last_undegraded":"2024-04-25T09:29:04.608000+0000","last_fullsized":"2024-04-25T09:29:04.608000+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.4","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.551457+0000","last_change":"2024-04-25T09:28:56.555285+0000","last_active":"2024-04-25T09:29:04.551457+0000","last_peered":"2024-04-25T09:29:04.551457+0000","last_clean":"2024-04-25T09:29:04.551457+0000","last_became_active":"2024-04-25T09:28:56.546535+0000","last_became_peered":"2024-04-25T09:28:56.546535+0000","last_unstale":"2024-04-25T09:29:04.551457+0000","last_undegraded":"2024-04-25T09:29:04.551457+0000","last_fullsized":"2024-04-25T09:29:04.551457+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.3","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.610411+0000","last_change":"2024-04-25T09:28:56.555708+0000","last_active":"2024-04-25T09:29:04.610411+0000","last_peered":"2024-04-25T09:29:04.610411+0000","last_clean":"2024-04-25T09:29:04.610411+0000","last_became_active":"2024-04-25T09:28:56.547591+0000","last_became_peered":"2024-04-25T09:28:56.547591+0000","last_unstale":"2024-04-25T09:29:04.610411+0000","last_undegraded":"2024-04-25T09:29:04.610411+0000","last_fullsized":"2024-04-25T09:29:04.610411+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.2","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.533005+0000","last_change":"2024-04-25T09:28:56.538188+0000","last_active":"2024-04-25T09:29:04.533005+0000","last_peered":"2024-04-25T09:29:04.533005+0000","last_clean":"2024-04-25T09:29:04.533005+0000","last_became_active":"2024-04-25T09:28:56.531302+0000","last_became_peered":"2024-04-25T09:28:56.531302+0000","last_unstale":"2024-04-25T09:29:04.533005+0000","last_undegraded":"2024-04-25T09:29:04.533005+0000","last_fullsized":"2024-04-25T09:29:04.533005+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.1","version":"0'0","reported_seq":25,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.504442+0000","last_change":"2024-04-25T09:28:56.560898+0000","last_active":"2024-04-25T09:29:04.504442+0000","last_peered":"2024-04-25T09:29:04.504442+0000","last_clean":"2024-04-25T09:29:04.504442+0000","last_became_active":"2024-04-25T09:28:56.551023+0000","last_became_peered":"2024-04-25T09:28:56.551023+0000","last_unstale":"2024-04-25T09:29:04.504442+0000","last_undegraded":"2024-04-25T09:29:04.504442+0000","last_fullsized":"2024-04-25T09:29:04.504442+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.0","version":"0'0","reported_seq":29,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.611781+0000","last_change":"2024-04-25T09:28:56.576712+0000","last_active":"2024-04-25T09:29:04.611781+0000","last_peered":"2024-04-25T09:29:04.611781+0000","last_clean":"2024-04-25T09:29:04.611781+0000","last_became_active":"2024-04-25T09:28:56.561446+0000","last_became_peered":"2024-04-25T09:28:56.561446+0000","last_unstale":"2024-04-25T09:29:04.611781+0000","last_undegraded":"2024-04-25T09:29:04.611781+0000","last_fullsized":"2024-04-25T09:29:04.611781+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:28:55.381331+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":843,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.610832+0000","last_change":"2024-04-25T09:23:24.588688+0000","last_active":"2024-04-25T09:29:04.610832+0000","last_peered":"2024-04-25T09:29:04.610832+0000","last_clean":"2024-04-25T09:29:04.610832+0000","last_became_active":"2024-04-25T09:23:24.583493+0000","last_became_peered":"2024-04-25T09:23:24.583493+0000","last_unstale":"2024-04-25T09:29:04.610832+0000","last_undegraded":"2024-04-25T09:29:04.610832+0000","last_fullsized":"2024-04-25T09:29:04.610832+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:23.538402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:23.538402+0000","last_clean_scrub_stamp":"2024-04-25T09:23:23.538402+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.558713+0000","last_change":"2024-04-25T09:23:27.662397+0000","last_active":"2024-04-25T09:29:04.558713+0000","last_peered":"2024-04-25T09:29:04.558713+0000","last_clean":"2024-04-25T09:29:04.558713+0000","last_became_active":"2024-04-25T09:23:27.654208+0000","last_became_peered":"2024-04-25T09:23:27.654208+0000","last_unstale":"2024-04-25T09:29:04.558713+0000","last_undegraded":"2024-04-25T09:29:04.558713+0000","last_fullsized":"2024-04-25T09:29:04.558713+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":705,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.633547+0000","last_change":"2024-04-25T09:23:27.657105+0000","last_active":"2024-04-25T09:29:04.633547+0000","last_peered":"2024-04-25T09:29:04.633547+0000","last_clean":"2024-04-25T09:29:04.633547+0000","last_became_active":"2024-04-25T09:23:27.652630+0000","last_became_peered":"2024-04-25T09:23:27.652630+0000","last_unstale":"2024-04-25T09:29:04.633547+0000","last_undegraded":"2024-04-25T09:29:04.633547+0000","last_fullsized":"2024-04-25T09:29:04.633547+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.544980+0000","last_change":"2024-04-25T09:23:27.635085+0000","last_active":"2024-04-25T09:29:04.544980+0000","last_peered":"2024-04-25T09:29:04.544980+0000","last_clean":"2024-04-25T09:29:04.544980+0000","last_became_active":"2024-04-25T09:23:27.629956+0000","last_became_peered":"2024-04-25T09:23:27.629956+0000","last_unstale":"2024-04-25T09:29:04.544980+0000","last_undegraded":"2024-04-25T09:29:04.544980+0000","last_fullsized":"2024-04-25T09:29:04.544980+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":705,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.615186+0000","last_change":"2024-04-25T09:23:27.642661+0000","last_active":"2024-04-25T09:29:04.615186+0000","last_peered":"2024-04-25T09:29:04.615186+0000","last_clean":"2024-04-25T09:29:04.615186+0000","last_became_active":"2024-04-25T09:23:27.635754+0000","last_became_peered":"2024-04-25T09:23:27.635754+0000","last_unstale":"2024-04-25T09:29:04.615186+0000","last_undegraded":"2024-04-25T09:29:04.615186+0000","last_fullsized":"2024-04-25T09:29:04.615186+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":697,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.605955+0000","last_change":"2024-04-25T09:23:27.631860+0000","last_active":"2024-04-25T09:29:04.605955+0000","last_peered":"2024-04-25T09:29:04.605955+0000","last_clean":"2024-04-25T09:29:04.605955+0000","last_became_active":"2024-04-25T09:23:27.627575+0000","last_became_peered":"2024-04-25T09:23:27.627575+0000","last_unstale":"2024-04-25T09:29:04.605955+0000","last_undegraded":"2024-04-25T09:29:04.605955+0000","last_fullsized":"2024-04-25T09:29:04.605955+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":707,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.655721+0000","last_change":"2024-04-25T09:23:27.641008+0000","last_active":"2024-04-25T09:29:04.655721+0000","last_peered":"2024-04-25T09:29:04.655721+0000","last_clean":"2024-04-25T09:29:04.655721+0000","last_became_active":"2024-04-25T09:23:27.635511+0000","last_became_peered":"2024-04-25T09:23:27.635511+0000","last_unstale":"2024-04-25T09:29:04.655721+0000","last_undegraded":"2024-04-25T09:29:04.655721+0000","last_fullsized":"2024-04-25T09:29:04.655721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":697,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.613085+0000","last_change":"2024-04-25T09:23:27.675576+0000","last_active":"2024-04-25T09:29:04.613085+0000","last_peered":"2024-04-25T09:29:04.613085+0000","last_clean":"2024-04-25T09:29:04.613085+0000","last_became_active":"2024-04-25T09:23:27.671371+0000","last_became_peered":"2024-04-25T09:23:27.671371+0000","last_unstale":"2024-04-25T09:29:04.613085+0000","last_undegraded":"2024-04-25T09:29:04.613085+0000","last_fullsized":"2024-04-25T09:29:04.613085+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-25T09:29:04.528911+0000","last_change":"2024-04-25T09:23:27.667733+0000","last_active":"2024-04-25T09:29:04.528911+0000","last_peered":"2024-04-25T09:29:04.528911+0000","last_clean":"2024-04-25T09:29:04.528911+0000","last_became_active":"2024-04-25T09:23:27.662121+0000","last_became_peered":"2024-04-25T09:23:27.662121+0000","last_unstale":"2024-04-25T09:29:04.528911+0000","last_undegraded":"2024-04-25T09:29:04.528911+0000","last_fullsized":"2024-04-25T09:29:04.528911+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:23:26.552349+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":72,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36780,"kb_used_data":1508,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104820820,"statfs":{"total":107374182400,"available":107336519680,"internally_reserved":0,"allocated":1544192,"data_stored":895709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36196,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104821404,"statfs":{"total":107374182400,"available":107337117696,"internally_reserved":0,"allocated":946176,"data_stored":305338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36196,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104821404,"statfs":{"total":107374182400,"available":107337117696,"internally_reserved":0,"allocated":946176,"data_stored":305338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":40940,"kb_used_data":1508,"kb_used_omap":0,"kb_used_meta":39424,"kb_avail":104816660,"statfs":{"total":107374182400,"available":107332259840,"internally_reserved":0,"allocated":1544192,"data_stored":895710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40370176},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T09:29:12.778 INFO:tasks.ceph:pgid 72.f last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.778 INFO:tasks.ceph:pgid 72.e last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.779 INFO:tasks.ceph:pgid 72.d last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.779 INFO:tasks.ceph:pgid 72.c last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.779 INFO:tasks.ceph:pgid 72.b last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.779 INFO:tasks.ceph:pgid 72.a last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.779 INFO:tasks.ceph:pgid 72.9 last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.779 INFO:tasks.ceph:pgid 72.8 last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.779 INFO:tasks.ceph:pgid 72.7 last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.779 INFO:tasks.ceph:pgid 72.6 last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.780 INFO:tasks.ceph:pgid 72.5 last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.780 INFO:tasks.ceph:pgid 72.4 last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.780 INFO:tasks.ceph:pgid 72.3 last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.780 INFO:tasks.ceph:pgid 72.2 last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.780 INFO:tasks.ceph:pgid 72.1 last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.780 INFO:tasks.ceph:pgid 72.0 last_scrub_stamp 2024-04-25T09:28:55.381331+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=28, tm_sec=55, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.780 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-25T09:23:23.538402+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=23, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.780 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-25T09:23:26.552349+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=23, tm_sec=26, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.780 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-25T09:23:26.552349+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=23, tm_sec=26, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.781 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-25T09:23:26.552349+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=23, tm_sec=26, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.781 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-25T09:23:26.552349+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=23, tm_sec=26, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.781 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-25T09:23:26.552349+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=23, tm_sec=26, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.781 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-25T09:23:26.552349+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=23, tm_sec=26, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.781 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-25T09:23:26.552349+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=23, tm_sec=26, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.781 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-25T09:23:26.552349+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=23, tm_sec=26, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=9, tm_min=29, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T09:29:12.781 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T09:29:32.782 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T09:29:32.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:32.919+0000 7fe05dcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:32.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:32.928+0000 7fe05dcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:33.109 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T09:29:33.109 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T09:29:33.121 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-04-25T09:29:31.580805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":150192,"kb_used_data":4944,"kb_used_omap":0,"kb_used_meta":145216,"kb_avail":419280208,"statfs":{"total":429496729600,"available":429342932992,"internally_reserved":0,"allocated":5062656,"data_stored":2430527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":148701184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002893"},"pg_stats":[{"pgid":"72.f","version":"0'0","reported_seq":33,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:10.994350+0000","last_change":"2024-04-25T09:29:11.095316+0000","last_active":"2024-04-25T09:29:10.994350+0000","last_peered":"2024-04-25T09:29:10.994350+0000","last_clean":"2024-04-25T09:29:10.994350+0000","last_became_active":"2024-04-25T09:28:56.594281+0000","last_became_peered":"2024-04-25T09:28:56.594281+0000","last_unstale":"2024-04-25T09:29:10.994350+0000","last_undegraded":"2024-04-25T09:29:10.994350+0000","last_fullsized":"2024-04-25T09:29:10.994350+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.095143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.095143+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.e","version":"0'0","reported_seq":31,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:11.967494+0000","last_change":"2024-04-25T09:29:11.978439+0000","last_active":"2024-04-25T09:29:11.967494+0000","last_peered":"2024-04-25T09:29:11.967494+0000","last_clean":"2024-04-25T09:29:11.967494+0000","last_became_active":"2024-04-25T09:28:56.581636+0000","last_became_peered":"2024-04-25T09:28:56.581636+0000","last_unstale":"2024-04-25T09:29:11.967494+0000","last_undegraded":"2024-04-25T09:29:11.967494+0000","last_fullsized":"2024-04-25T09:29:11.967494+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.978253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.978253+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.d","version":"0'0","reported_seq":33,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:10.996989+0000","last_change":"2024-04-25T09:29:11.096722+0000","last_active":"2024-04-25T09:29:10.996989+0000","last_peered":"2024-04-25T09:29:10.996989+0000","last_clean":"2024-04-25T09:29:10.996989+0000","last_became_active":"2024-04-25T09:28:56.569011+0000","last_became_peered":"2024-04-25T09:28:56.569011+0000","last_unstale":"2024-04-25T09:29:10.996989+0000","last_undegraded":"2024-04-25T09:29:10.996989+0000","last_fullsized":"2024-04-25T09:29:10.996989+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.096558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.096558+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.c","version":"0'0","reported_seq":37,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:12.455417+0000","last_change":"2024-04-25T09:29:12.476872+0000","last_active":"2024-04-25T09:29:12.455417+0000","last_peered":"2024-04-25T09:29:12.455417+0000","last_clean":"2024-04-25T09:29:12.455417+0000","last_became_active":"2024-04-25T09:28:56.566867+0000","last_became_peered":"2024-04-25T09:28:56.566867+0000","last_unstale":"2024-04-25T09:29:12.455417+0000","last_undegraded":"2024-04-25T09:29:12.455417+0000","last_fullsized":"2024-04-25T09:29:12.455417+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:12.476644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:12.476644+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.b","version":"0'0","reported_seq":33,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:10.988753+0000","last_change":"2024-04-25T09:29:11.066052+0000","last_active":"2024-04-25T09:29:10.988753+0000","last_peered":"2024-04-25T09:29:10.988753+0000","last_clean":"2024-04-25T09:29:10.988753+0000","last_became_active":"2024-04-25T09:28:56.625254+0000","last_became_peered":"2024-04-25T09:28:56.625254+0000","last_unstale":"2024-04-25T09:29:10.988753+0000","last_undegraded":"2024-04-25T09:29:10.988753+0000","last_fullsized":"2024-04-25T09:29:10.988753+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.065881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.065881+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.a","version":"0'0","reported_seq":37,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:12.452577+0000","last_change":"2024-04-25T09:29:12.474741+0000","last_active":"2024-04-25T09:29:12.452577+0000","last_peered":"2024-04-25T09:29:12.452577+0000","last_clean":"2024-04-25T09:29:12.452577+0000","last_became_active":"2024-04-25T09:28:56.556905+0000","last_became_peered":"2024-04-25T09:28:56.556905+0000","last_unstale":"2024-04-25T09:29:12.452577+0000","last_undegraded":"2024-04-25T09:29:12.452577+0000","last_fullsized":"2024-04-25T09:29:12.452577+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:12.474547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:12.474547+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.9","version":"0'0","reported_seq":33,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:10.987191+0000","last_change":"2024-04-25T09:29:11.092391+0000","last_active":"2024-04-25T09:29:10.987191+0000","last_peered":"2024-04-25T09:29:10.987191+0000","last_clean":"2024-04-25T09:29:10.987191+0000","last_became_active":"2024-04-25T09:28:56.574219+0000","last_became_peered":"2024-04-25T09:28:56.574219+0000","last_unstale":"2024-04-25T09:29:10.987191+0000","last_undegraded":"2024-04-25T09:29:10.987191+0000","last_fullsized":"2024-04-25T09:29:10.987191+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.092212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.092212+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.8","version":"0'0","reported_seq":31,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:11.966470+0000","last_change":"2024-04-25T09:29:11.980091+0000","last_active":"2024-04-25T09:29:11.966470+0000","last_peered":"2024-04-25T09:29:11.966470+0000","last_clean":"2024-04-25T09:29:11.966470+0000","last_became_active":"2024-04-25T09:28:56.620437+0000","last_became_peered":"2024-04-25T09:28:56.620437+0000","last_unstale":"2024-04-25T09:29:11.966470+0000","last_undegraded":"2024-04-25T09:29:11.966470+0000","last_fullsized":"2024-04-25T09:29:11.966470+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.979923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.979923+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.7","version":"0'0","reported_seq":37,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:12.451615+0000","last_change":"2024-04-25T09:29:12.472773+0000","last_active":"2024-04-25T09:29:12.451615+0000","last_peered":"2024-04-25T09:29:12.451615+0000","last_clean":"2024-04-25T09:29:12.451615+0000","last_became_active":"2024-04-25T09:28:56.532134+0000","last_became_peered":"2024-04-25T09:28:56.532134+0000","last_unstale":"2024-04-25T09:29:12.451615+0000","last_undegraded":"2024-04-25T09:29:12.451615+0000","last_fullsized":"2024-04-25T09:29:12.451615+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:12.472592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:12.472592+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.6","version":"0'0","reported_seq":33,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:10.984868+0000","last_change":"2024-04-25T09:29:11.063796+0000","last_active":"2024-04-25T09:29:10.984868+0000","last_peered":"2024-04-25T09:29:10.984868+0000","last_clean":"2024-04-25T09:29:10.984868+0000","last_became_active":"2024-04-25T09:28:56.584773+0000","last_became_peered":"2024-04-25T09:28:56.584773+0000","last_unstale":"2024-04-25T09:29:10.984868+0000","last_undegraded":"2024-04-25T09:29:10.984868+0000","last_fullsized":"2024-04-25T09:29:10.984868+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.063619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.063619+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.5","version":"0'0","reported_seq":37,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:12.449844+0000","last_change":"2024-04-25T09:29:12.467225+0000","last_active":"2024-04-25T09:29:12.449844+0000","last_peered":"2024-04-25T09:29:12.449844+0000","last_clean":"2024-04-25T09:29:12.449844+0000","last_became_active":"2024-04-25T09:28:56.579671+0000","last_became_peered":"2024-04-25T09:28:56.579671+0000","last_unstale":"2024-04-25T09:29:12.449844+0000","last_undegraded":"2024-04-25T09:29:12.449844+0000","last_fullsized":"2024-04-25T09:29:12.449844+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:12.467039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:12.467039+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.4","version":"0'0","reported_seq":33,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:11.474422+0000","last_change":"2024-04-25T09:29:11.485527+0000","last_active":"2024-04-25T09:29:11.474422+0000","last_peered":"2024-04-25T09:29:11.474422+0000","last_clean":"2024-04-25T09:29:11.474422+0000","last_became_active":"2024-04-25T09:28:56.546535+0000","last_became_peered":"2024-04-25T09:28:56.546535+0000","last_unstale":"2024-04-25T09:29:11.474422+0000","last_undegraded":"2024-04-25T09:29:11.474422+0000","last_fullsized":"2024-04-25T09:29:11.474422+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.485346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.485346+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.3","version":"0'0","reported_seq":33,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:10.983050+0000","last_change":"2024-04-25T09:29:11.093873+0000","last_active":"2024-04-25T09:29:10.983050+0000","last_peered":"2024-04-25T09:29:10.983050+0000","last_clean":"2024-04-25T09:29:10.983050+0000","last_became_active":"2024-04-25T09:28:56.547591+0000","last_became_peered":"2024-04-25T09:28:56.547591+0000","last_unstale":"2024-04-25T09:29:10.983050+0000","last_undegraded":"2024-04-25T09:29:10.983050+0000","last_fullsized":"2024-04-25T09:29:10.983050+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.093708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.093708+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.2","version":"0'0","reported_seq":33,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:11.472529+0000","last_change":"2024-04-25T09:29:11.484512+0000","last_active":"2024-04-25T09:29:11.472529+0000","last_peered":"2024-04-25T09:29:11.472529+0000","last_clean":"2024-04-25T09:29:11.472529+0000","last_became_active":"2024-04-25T09:28:56.531302+0000","last_became_peered":"2024-04-25T09:28:56.531302+0000","last_unstale":"2024-04-25T09:29:11.472529+0000","last_undegraded":"2024-04-25T09:29:11.472529+0000","last_fullsized":"2024-04-25T09:29:11.472529+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.484341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.484341+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.1","version":"0'0","reported_seq":33,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:11.471253+0000","last_change":"2024-04-25T09:29:11.481190+0000","last_active":"2024-04-25T09:29:11.471253+0000","last_peered":"2024-04-25T09:29:11.471253+0000","last_clean":"2024-04-25T09:29:11.471253+0000","last_became_active":"2024-04-25T09:28:56.551023+0000","last_became_peered":"2024-04-25T09:28:56.551023+0000","last_unstale":"2024-04-25T09:29:11.471253+0000","last_undegraded":"2024-04-25T09:29:11.471253+0000","last_fullsized":"2024-04-25T09:29:11.471253+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.480989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.480989+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.0","version":"0'0","reported_seq":37,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:12.448878+0000","last_change":"2024-04-25T09:29:12.469638+0000","last_active":"2024-04-25T09:29:12.448878+0000","last_peered":"2024-04-25T09:29:12.448878+0000","last_clean":"2024-04-25T09:29:12.448878+0000","last_became_active":"2024-04-25T09:28:56.561446+0000","last_became_peered":"2024-04-25T09:28:56.561446+0000","last_unstale":"2024-04-25T09:29:12.448878+0000","last_undegraded":"2024-04-25T09:29:12.448878+0000","last_fullsized":"2024-04-25T09:29:12.448878+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:12.469432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:12.469432+0000","last_clean_scrub_stamp":"2024-04-25T09:28:55.381331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":851,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:12.447251+0000","last_change":"2024-04-25T09:29:12.478936+0000","last_active":"2024-04-25T09:29:12.447251+0000","last_peered":"2024-04-25T09:29:12.447251+0000","last_clean":"2024-04-25T09:29:12.447251+0000","last_became_active":"2024-04-25T09:23:24.583493+0000","last_became_peered":"2024-04-25T09:23:24.583493+0000","last_unstale":"2024-04-25T09:29:12.447251+0000","last_undegraded":"2024-04-25T09:29:12.447251+0000","last_fullsized":"2024-04-25T09:29:12.447251+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'97","last_scrub_stamp":"2024-04-25T09:29:12.478733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:12.478733+0000","last_clean_scrub_stamp":"2024-04-25T09:23:23.538402+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":709,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:11.470765+0000","last_change":"2024-04-25T09:29:11.483182+0000","last_active":"2024-04-25T09:29:11.470765+0000","last_peered":"2024-04-25T09:29:11.470765+0000","last_clean":"2024-04-25T09:29:11.470765+0000","last_became_active":"2024-04-25T09:23:27.654208+0000","last_became_peered":"2024-04-25T09:23:27.654208+0000","last_unstale":"2024-04-25T09:29:11.470765+0000","last_undegraded":"2024-04-25T09:29:11.470765+0000","last_fullsized":"2024-04-25T09:29:11.470765+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.482952+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":713,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:10.981725+0000","last_change":"2024-04-25T09:29:11.065395+0000","last_active":"2024-04-25T09:29:10.981725+0000","last_peered":"2024-04-25T09:29:10.981725+0000","last_clean":"2024-04-25T09:29:10.981725+0000","last_became_active":"2024-04-25T09:23:27.652630+0000","last_became_peered":"2024-04-25T09:23:27.652630+0000","last_unstale":"2024-04-25T09:29:10.981725+0000","last_undegraded":"2024-04-25T09:29:10.981725+0000","last_fullsized":"2024-04-25T09:29:10.981725+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.065221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.065221+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":709,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:11.469479+0000","last_change":"2024-04-25T09:29:11.483007+0000","last_active":"2024-04-25T09:29:11.469479+0000","last_peered":"2024-04-25T09:29:11.469479+0000","last_clean":"2024-04-25T09:29:11.469479+0000","last_became_active":"2024-04-25T09:23:27.629956+0000","last_became_peered":"2024-04-25T09:23:27.629956+0000","last_unstale":"2024-04-25T09:29:11.469479+0000","last_undegraded":"2024-04-25T09:29:11.469479+0000","last_fullsized":"2024-04-25T09:29:11.469479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.482819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.482819+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":713,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:10.982042+0000","last_change":"2024-04-25T09:29:11.090828+0000","last_active":"2024-04-25T09:29:10.982042+0000","last_peered":"2024-04-25T09:29:10.982042+0000","last_clean":"2024-04-25T09:29:10.982042+0000","last_became_active":"2024-04-25T09:23:27.635754+0000","last_became_peered":"2024-04-25T09:23:27.635754+0000","last_unstale":"2024-04-25T09:29:10.982042+0000","last_undegraded":"2024-04-25T09:29:10.982042+0000","last_fullsized":"2024-04-25T09:29:10.982042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.090651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.090651+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":705,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:11.965136+0000","last_change":"2024-04-25T09:29:11.976037+0000","last_active":"2024-04-25T09:29:11.965136+0000","last_peered":"2024-04-25T09:29:11.965136+0000","last_clean":"2024-04-25T09:29:11.965136+0000","last_became_active":"2024-04-25T09:23:27.627575+0000","last_became_peered":"2024-04-25T09:23:27.627575+0000","last_unstale":"2024-04-25T09:29:11.965136+0000","last_undegraded":"2024-04-25T09:29:11.965136+0000","last_fullsized":"2024-04-25T09:29:11.965136+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.975854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.975854+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":715,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:12.447242+0000","last_change":"2024-04-25T09:29:12.466960+0000","last_active":"2024-04-25T09:29:12.447242+0000","last_peered":"2024-04-25T09:29:12.447242+0000","last_clean":"2024-04-25T09:29:12.447242+0000","last_became_active":"2024-04-25T09:23:27.635511+0000","last_became_peered":"2024-04-25T09:23:27.635511+0000","last_unstale":"2024-04-25T09:29:12.447242+0000","last_undegraded":"2024-04-25T09:29:12.447242+0000","last_fullsized":"2024-04-25T09:29:12.447242+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:12.466651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:12.466651+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":705,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:11.966664+0000","last_change":"2024-04-25T09:29:11.978025+0000","last_active":"2024-04-25T09:29:11.966664+0000","last_peered":"2024-04-25T09:29:11.966664+0000","last_clean":"2024-04-25T09:29:11.966664+0000","last_became_active":"2024-04-25T09:23:27.671371+0000","last_became_peered":"2024-04-25T09:23:27.671371+0000","last_unstale":"2024-04-25T09:29:11.966664+0000","last_undegraded":"2024-04-25T09:29:11.966664+0000","last_fullsized":"2024-04-25T09:29:11.966664+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.977849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.977849+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":709,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T09:29:11.472283+0000","last_change":"2024-04-25T09:29:11.484832+0000","last_active":"2024-04-25T09:29:11.472283+0000","last_peered":"2024-04-25T09:29:11.472283+0000","last_clean":"2024-04-25T09:29:11.472283+0000","last_became_active":"2024-04-25T09:23:27.662121+0000","last_became_peered":"2024-04-25T09:23:27.662121+0000","last_unstale":"2024-04-25T09:29:11.472283+0000","last_undegraded":"2024-04-25T09:29:11.472283+0000","last_fullsized":"2024-04-25T09:29:11.472283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T09:29:11.484660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T09:29:11.484660+0000","last_clean_scrub_stamp":"2024-04-25T09:23:26.552349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":72,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36800,"kb_used_data":1528,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104820800,"statfs":{"total":107374182400,"available":107336499200,"internally_reserved":0,"allocated":1564672,"data_stored":902817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36216,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104821384,"statfs":{"total":107374182400,"available":107337097216,"internally_reserved":0,"allocated":966656,"data_stored":312446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36216,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":104821384,"statfs":{"total":107374182400,"available":107337097216,"internally_reserved":0,"allocated":966656,"data_stored":312446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":40960,"kb_used_data":1528,"kb_used_omap":0,"kb_used_meta":39424,"kb_avail":104816640,"statfs":{"total":107374182400,"available":107332239360,"internally_reserved":0,"allocated":1564672,"data_stored":902818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40370176},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":72,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T09:29:33.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-25T09:29:33.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:33.241+0000 7f69b2eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:33.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-25T09:29:33.253+0000 7f69b2eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-25T09:29:33.457 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T09:29:33.457 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T09:29:33.457 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-25T09:29:33.457 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T09:29:33.524 INFO:tasks.ceph.osd.0:Stopped 2024-04-25T09:29:33.525 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-25T09:29:33.525 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T09:29:33.596 INFO:tasks.ceph.osd.1:Stopped 2024-04-25T09:29:33.596 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-25T09:29:33.596 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T09:29:33.664 INFO:tasks.ceph.osd.2:Stopped 2024-04-25T09:29:33.664 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-25T09:29:33.664 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T09:29:33.736 INFO:tasks.ceph.osd.3:Stopped 2024-04-25T09:29:33.736 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T09:29:33.736 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-25T09:29:33.736 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T09:29:33.762 INFO:tasks.ceph.mgr.x:Stopped 2024-04-25T09:29:33.763 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T09:29:33.763 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-25T09:29:33.763 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T09:29:33.775 INFO:tasks.ceph.mon.a:Stopped 2024-04-25T09:29:33.775 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T09:29:33.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-25T09:29:33.850 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-25T09:29:33.850 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-25T09:29:34.045 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-25T09:29:34.045 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-25T09:29:34.152 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi164.front.sepia.ceph.com 2024-04-25T09:29:34.153 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-25T09:29:34.340 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi164.front.sepia.ceph.com 2024-04-25T09:29:34.340 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-25T09:29:34.406 INFO:tasks.ceph:Archiving mon data... 2024-04-25T09:29:34.408 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi/7672977/data/mon.a.tgz 2024-04-25T09:29:34.408 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-04-25T09:29:34.423 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.J9Ke8FPWsz 2024-04-25T09:29:34.423 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.J9Ke8FPWsz 2024-04-25T09:29:34.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.J9Ke8FPWsz 2024-04-25T09:29:34.690 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.J9Ke8FPWsz is 365KB 2024-04-25T09:29:34.786 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.J9Ke8FPWsz 2024-04-25T09:29:34.801 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T09:29:34.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T09:29:34.844 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T09:29:34.921 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T09:29:34.938 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T09:29:34.938 INFO:tasks.ceph:Archiving crash dumps... 2024-04-25T09:29:34.939 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi/7672977/remote/smithi043/crash 2024-04-25T09:29:34.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T09:29:34.987 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi/7672977/remote/smithi164/crash 2024-04-25T09:29:34.988 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T09:29:35.023 INFO:tasks.ceph:Compressing logs... 2024-04-25T09:29:35.023 DEBUG:teuthology.orchestra.run.smithi043:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T09:29:35.028 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T09:29:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27470.log 2024-04-25T09:29:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29937.log 2024-04-25T09:29:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29895.log 2024-04-25T09:29:35.066 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27470.log.gz 2024-04-25T09:29:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25989.log 2024-04-25T09:29:35.066 INFO:teuthology.orchestra.run.smithi043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29937.log.gz 2024-04-25T09:29:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log/var/log/ceph/ceph-client.admin.29895.log: 2024-04-25T09:29:35.066 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29895.log.gz 2024-04-25T09:29:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.25989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29811.log 2024-04-25T09:29:35.066 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.25989.log.gz 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27320.log 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28094.log 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29811.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29811.log.gz 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27638.log 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27320.log.gz 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29727.log 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27638.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr: 93.7% -- replaced with /var/log/ceph/ceph-client.admin.28094.log.gz 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27638.log.gz 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27045.log 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29727.log.gz 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27179.log 2024-04-25T09:29:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27851.log 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27045.log.gz 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27179.log: gzip 88.5% -5 -- replaced with /var/log/ceph/ceph.audit.log.gz --verbose 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr: -- /var/log/ceph/ceph-client.admin.30021.log 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27179.log.gz 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27661.log 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27851.log.gz 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27809.log 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30021.log.gz 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30063.log 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27661.log.gz 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29685.log 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27809.log.gz 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27323.log 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30063.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30063.log.gz 2024-04-25T09:29:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-25T09:29:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29685.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29685.log.gz 2024-04-25T09:29:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27323.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-25T09:29:35.069 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27323.log.gz 2024-04-25T09:29:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27767.log 2024-04-25T09:29:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27639.log 2024-04-25T09:29:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.27767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30230.log 2024-04-25T09:29:35.069 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27767.log.gz 2024-04-25T09:29:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27321.log 2024-04-25T09:29:35.069 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27639.log.gz 2024-04-25T09:29:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30230.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-25T09:29:35.069 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30230.log.gz 2024-04-25T09:29:35.070 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27599.log 2024-04-25T09:29:35.070 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27321.log.gz 2024-04-25T09:29:35.070 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30105.log 2024-04-25T09:29:35.070 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29769.log 2024-04-25T09:29:35.070 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27599.log.gz 2024-04-25T09:29:35.070 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30105.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.23588.log 2024-04-25T09:29:35.070 INFO:teuthology.orchestra.run.smithi043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30105.log.gz 2024-04-25T09:29:35.070 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26989.log 2024-04-25T09:29:35.070 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29769.log.gz 2024-04-25T09:29:35.070 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph.tmp-client.admin.23588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27471.log 2024-04-25T09:29:35.070 INFO:teuthology.orchestra.run.smithi043.stderr: 67.2% -- replaced with /var/log/ceph/ceph.tmp-client.admin.23588.log.gz 2024-04-25T09:29:35.070 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.26989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27095.log 2024-04-25T09:29:35.070 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.26989.log.gz 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27322.log 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27471.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27471.log.gz 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27221.log 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.27095.log.gz 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30147.log 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27322.log.gz 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27137.log 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27221.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27221.log.gz 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27493.log 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30147.log.gz 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29643.log 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27137.log.gz 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25910.log 2024-04-25T09:29:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27493.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27493.log.gz 2024-04-25T09:29:35.072 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29853.log 2024-04-25T09:29:35.072 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29643.log.gz 2024-04-25T09:29:35.072 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.25910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26174.log 2024-04-25T09:29:35.072 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.25910.log.gz 2024-04-25T09:29:35.072 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27431.log 2024-04-25T09:29:35.072 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29853.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29853.log.gz 2024-04-25T09:29:35.072 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.26174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26145.log 2024-04-25T09:29:35.072 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.26174.log.gz 2024-04-25T09:29:35.072 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29979.log 2024-04-25T09:29:35.072 INFO:teuthology.orchestra.run.smithi043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27431.log.gz 2024-04-25T09:29:35.073 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.26145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30190.log 2024-04-25T09:29:35.073 INFO:teuthology.orchestra.run.smithi043.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.26145.log.gz 2024-04-25T09:29:35.073 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28069.log 2024-04-25T09:29:35.073 INFO:teuthology.orchestra.run.smithi043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29979.log.gz 2024-04-25T09:29:35.073 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30190.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30190.log.gz 2024-04-25T09:29:35.073 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28069.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28069.log.gz 2024-04-25T09:29:35.098 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25628.log 2024-04-25T09:29:35.098 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-25T09:29:35.100 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25548.log 2024-04-25T09:29:35.100 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.25628.log: /var/log/ceph/ceph-osd.2.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.25628.log.gz 2024-04-25T09:29:35.100 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-25T09:29:35.100 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.25548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25588.log 2024-04-25T09:29:35.100 INFO:teuthology.orchestra.run.smithi164.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.25548.log.gz 2024-04-25T09:29:35.100 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-25T09:29:35.100 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.25588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25668.log 2024-04-25T09:29:35.100 INFO:teuthology.orchestra.run.smithi164.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.25588.log.gz 2024-04-25T09:29:35.100 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.25668.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.25668.log.gz 2024-04-25T09:29:35.143 INFO:teuthology.orchestra.run.smithi164.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-25T09:29:35.333 INFO:teuthology.orchestra.run.smithi043.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-25T09:29:36.236 INFO:teuthology.orchestra.run.smithi164.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-25T09:29:36.271 INFO:teuthology.orchestra.run.smithi164.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-25T09:29:36.273 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-04-25T09:29:36.274 INFO:teuthology.orchestra.run.smithi164.stderr:real 0m1.185s 2024-04-25T09:29:36.274 INFO:teuthology.orchestra.run.smithi164.stderr:user 0m2.264s 2024-04-25T09:29:36.274 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m0.110s 2024-04-25T09:29:36.285 INFO:teuthology.orchestra.run.smithi043.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-25T09:29:36.377 INFO:teuthology.orchestra.run.smithi043.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-25T09:29:36.379 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-25T09:29:36.379 INFO:teuthology.orchestra.run.smithi043.stderr:real 0m1.339s 2024-04-25T09:29:36.379 INFO:teuthology.orchestra.run.smithi043.stderr:user 0m2.749s 2024-04-25T09:29:36.379 INFO:teuthology.orchestra.run.smithi043.stderr:sys 0m0.132s 2024-04-25T09:29:36.379 INFO:tasks.ceph:Archiving logs... 2024-04-25T09:29:36.379 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi/7672977/remote/smithi043/log 2024-04-25T09:29:36.381 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T09:29:36.901 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi/7672977/remote/smithi164/log 2024-04-25T09:29:36.902 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T09:29:37.311 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T09:29:37.321 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T09:29:37.321 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T09:29:37.324 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T09:29:37.407 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T09:29:37.417 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T09:29:37.417 DEBUG:teuthology.orchestra.run.smithi043:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T09:29:37.420 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T09:29:37.433 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-04-25T09:29:37.435 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-04-25T09:29:37.438 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T09:29:37.438 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-04-25T09:29:37.438 INFO:teuthology.orchestra.run.smithi043.stdout:^- hv01.front.sepia.ceph.com 2 6 377 22 +1759us[+1759us] +/- 683ms 2024-04-25T09:29:37.438 INFO:teuthology.orchestra.run.smithi043.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 -891us[ -905us] +/- 56ms 2024-04-25T09:29:37.439 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +1030us[+1030us] +/- 63ms 2024-04-25T09:29:37.439 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T09:29:37.440 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T09:29:37.440 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-04-25T09:29:37.440 INFO:teuthology.orchestra.run.smithi164.stdout:^- hv01.front.sepia.ceph.com 2 6 377 21 +1765us[+1782us] +/- 683ms 2024-04-25T09:29:37.440 INFO:teuthology.orchestra.run.smithi164.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -876us[ -859us] +/- 56ms 2024-04-25T09:29:37.440 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +1034us[+1051us] +/- 63ms 2024-04-25T09:29:37.440 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T09:29:37.441 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T09:29:37.450 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T09:29:37.450 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T09:29:37.460 DEBUG:teuthology.orchestra.run.smithi043:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T09:29:37.543 DEBUG:teuthology.orchestra.run.smithi164:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T09:29:37.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T09:29:37.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T09:29:37.631 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T09:29:37.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T09:29:37.658 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T09:29:37.667 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T09:29:37.681 INFO:teuthology.task.internal:Duration was 758.106624 seconds 2024-04-25T09:29:37.682 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T09:29:37.691 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T09:29:37.691 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T09:29:37.694 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T09:29:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T09:29:37.735 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T09:29:38.068 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T09:29:38.068 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi043.front.sepia.ceph.com 2024-04-25T09:29:38.068 DEBUG:teuthology.orchestra.run.smithi043:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T09:29:38.100 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2024-04-25T09:29:38.100 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T09:29:38.132 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T09:29:38.132 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T09:29:38.143 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T09:29:38.197 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T09:29:38.197 DEBUG:teuthology.orchestra.run.smithi043:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T09:29:38.211 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T09:29:38.327 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T09:29:38.337 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T09:29:38.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T09:29:38.362 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T09:29:38.385 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T09:29:38.395 DEBUG:teuthology.orchestra.run.smithi043:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:29:38.404 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:29:38.425 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = core 2024-04-25T09:29:38.447 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2024-04-25T09:29:38.465 DEBUG:teuthology.orchestra.run.smithi043:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:29:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T09:29:38.498 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:29:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T09:29:38.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T09:29:38.530 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T09:29:38.531 DEBUG:teuthology.misc:Transferring archived files from smithi043:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi/7672977/remote/smithi043 2024-04-25T09:29:38.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T09:29:38.572 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-04-25_08:07:11-crimson-rados-wip-matanb-crimson-testing-april-22-distro-crimson-smithi/7672977/remote/smithi164 2024-04-25T09:29:38.572 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T09:29:38.603 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T09:29:38.603 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T09:29:38.608 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T09:29:38.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T09:29:38.664 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T09:29:38.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T09:29:38.676 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T09:29:38.676 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T09:29:38.679 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T09:29:38.691 INFO:teuthology.orchestra.run.smithi043.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 09:29 /home/ubuntu/cephtest 2024-04-25T09:29:38.707 INFO:teuthology.orchestra.run.smithi164.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 09:29 /home/ubuntu/cephtest 2024-04-25T09:29:38.708 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T09:29:38.718 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T09:29:38.740 INFO:teuthology.run:Summary data: description: crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} duration: 758.1066238880157 flavor: crimson owner: scheduled_matan@teuthology success: true 2024-04-25T09:29:38.741 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T09:29:38.822 INFO:teuthology.run:pass