2024-02-14T17:27:29.700 INFO:root:teuthology version: 0.0.1.dev258+g9e44c16 2024-02-14T17:27:29.700 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809 --name gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi --verbose --description orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} --owner scheduled_gabrioux@teuthology -- /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809/orig.config.yaml 2024-02-14T17:27:29.742 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-14T17:27:29.820 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809 branch: wip-guits-testing-2024-02-14-1033 description: orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} email: gabrioux@ibm.com first_in_suite: false job_id: '7559809' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-testing-2024-02-14-1033 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 8727ce939fc044a98f215e29696bb1433f5758d1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8727ce939fc044a98f215e29696bb1433f5758d1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-testing-2024-02-14-1033 sha1: 8727ce939fc044a98f215e29696bb1433f5758d1 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 4718 sha1: 8727ce939fc044a98f215e29696bb1433f5758d1 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-testing-2024-02-14-1033 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_8727ce939fc044a98f215e29696bb1433f5758d1/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 8727ce939fc044a98f215e29696bb1433f5758d1 targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBRO4n+A/9PeQeviwJ8onInkR4bocXJa/AukutL0IlqcwSH5AO8WvTkxDW5eSyfThakbTjMVv2yuIAa6kdxoItI= smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGGEsMPMdo9+fZ0LxSlJIJJ1uEU6n+mFAeBWGFiPUe78hyRZb+CkOxV1dCgGRRpp0TMsu/Wb4Hs+1UyfneBZvC8= smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOvWGh6+0Ube1UCR24AGzdGs1VW3rag1Uoq9le34niBLnU5EcbLmWqFjCYG1evsX65Wvo8AVtnmz+47XTUk5ypM= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9e44c16382dfc4c612f7437cbb344c76223137cc timestamp: 2024-02-14_13:54:10 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-14T17:27:29.820 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_8727ce939fc044a98f215e29696bb1433f5758d1/qa; will attempt to use it 2024-02-14T17:27:29.821 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_8727ce939fc044a98f215e29696bb1433f5758d1/qa/tasks 2024-02-14T17:27:29.821 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-14T17:27:29.822 INFO:teuthology.task.internal:Checking packages... 2024-02-14T17:27:29.846 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8727ce939fc044a98f215e29696bb1433f5758d1' 2024-02-14T17:27:29.846 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-14T17:27:29.847 INFO:teuthology.packaging:ref: None 2024-02-14T17:27:29.847 INFO:teuthology.packaging:tag: None 2024-02-14T17:27:29.847 INFO:teuthology.packaging:branch: wip-guits-testing-2024-02-14-1033 2024-02-14T17:27:29.847 INFO:teuthology.packaging:sha1: 8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:27:29.847 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-testing-2024-02-14-1033 2024-02-14T17:27:30.203 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1430.g8727ce93 2024-02-14T17:27:30.205 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-14T17:27:30.212 INFO:teuthology.task.internal:no buildpackages task found 2024-02-14T17:27:30.212 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-14T17:27:30.226 INFO:teuthology.task.internal:Saving configuration 2024-02-14T17:27:30.238 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-14T17:27:30.244 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-14T17:27:30.268 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-14 17:21:36.119003', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBRO4n+A/9PeQeviwJ8onInkR4bocXJa/AukutL0IlqcwSH5AO8WvTkxDW5eSyfThakbTjMVv2yuIAa6kdxoItI='} 2024-02-14T17:27:30.290 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-14 17:21:36.120489', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGGEsMPMdo9+fZ0LxSlJIJJ1uEU6n+mFAeBWGFiPUe78hyRZb+CkOxV1dCgGRRpp0TMsu/Wb4Hs+1UyfneBZvC8='} 2024-02-14T17:27:30.312 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-14 17:21:36.121747', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOvWGh6+0Ube1UCR24AGzdGs1VW3rag1Uoq9le34niBLnU5EcbLmWqFjCYG1evsX65Wvo8AVtnmz+47XTUk5ypM='} 2024-02-14T17:27:30.312 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-14T17:27:30.318 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-14T17:27:30.319 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-02-14T17:27:30.319 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-02-14T17:27:30.319 INFO:teuthology.run_tasks:Running task console_log... 2024-02-14T17:27:30.483 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faffe87af70>, signals=[15]) 2024-02-14T17:27:30.483 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-14T17:27:30.490 INFO:teuthology.task.internal:Opening connections... 2024-02-14T17:27:30.490 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-02-14T17:27:30.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:27:30.597 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-02-14T17:27:30.598 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:27:30.703 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2024-02-14T17:27:30.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:27:30.791 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-14T17:27:30.799 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-02-14T17:27:30.818 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-02-14T17:27:30.818 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-02-14T17:27:30.876 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-02-14T17:27:30.876 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-02-14T17:27:30.876 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-02-14T17:27:30.876 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-02-14T17:27:30.876 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-02-14T17:27:30.876 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-02-14T17:27:30.876 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-14T17:27:30.876 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-02-14T17:27:30.877 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-02-14T17:27:30.877 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-14T17:27:30.877 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-02-14T17:27:30.877 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-14T17:27:30.877 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-14T17:27:30.877 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-14T17:27:30.877 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-02-14T17:27:30.900 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-02-14T17:27:30.918 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-02-14T17:27:30.918 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-02-14T17:27:30.977 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-02-14T17:27:30.977 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="9" 2024-02-14T17:27:30.977 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-02-14T17:27:30.977 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-02-14T17:27:30.977 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="9" 2024-02-14T17:27:30.977 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el9" 2024-02-14T17:27:30.977 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-14T17:27:30.977 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-02-14T17:27:30.977 INFO:teuthology.orchestra.run.smithi040.stdout:LOGO="fedora-logo-icon" 2024-02-14T17:27:30.977 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-14T17:27:30.978 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-02-14T17:27:30.978 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-14T17:27:30.978 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-14T17:27:30.978 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-14T17:27:30.978 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-02-14T17:27:30.999 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2024-02-14T17:27:31.017 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2024-02-14T17:27:31.017 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2024-02-14T17:27:31.075 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2024-02-14T17:27:31.075 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="9" 2024-02-14T17:27:31.075 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2024-02-14T17:27:31.075 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2024-02-14T17:27:31.075 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="9" 2024-02-14T17:27:31.075 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el9" 2024-02-14T17:27:31.076 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-14T17:27:31.076 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2024-02-14T17:27:31.076 INFO:teuthology.orchestra.run.smithi096.stdout:LOGO="fedora-logo-icon" 2024-02-14T17:27:31.076 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-14T17:27:31.076 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2024-02-14T17:27:31.076 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-14T17:27:31.076 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-14T17:27:31.076 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-14T17:27:31.076 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2024-02-14T17:27:31.091 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-14T17:27:31.100 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-14T17:27:31.105 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-14T17:27:31.105 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-02-14T17:27:31.107 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-02-14T17:27:31.110 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2024-02-14T17:27:31.132 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-14T17:27:31.138 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-14T17:27:31.138 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-02-14T17:27:31.165 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-02-14T17:27:31.167 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2024-02-14T17:27:31.293 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-14T17:27:31.371 INFO:teuthology.run_tasks:Running task kernel... 2024-02-14T17:27:31.387 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-14T17:27:31.387 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-14T17:27:31.388 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-14T17:27:31.388 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-14T17:27:31.389 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-14T17:27:31.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:27:31.406 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-02-14T17:27:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:27:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:27:31.408 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-02-14T17:27:31.408 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-02-14T17:27:31.463 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-361.el9.x86_64 2024-02-14T17:27:31.464 INFO:teuthology.task.kernel:Running kernel on smithi096: 5.14.0-361.el9.x86_64 2024-02-14T17:27:31.464 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum install -y kernel 2024-02-14T17:27:31.465 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-361.el9.x86_64 2024-02-14T17:27:31.465 INFO:teuthology.task.kernel:Running kernel on smithi040: 5.14.0-361.el9.x86_64 2024-02-14T17:27:31.465 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum install -y kernel 2024-02-14T17:27:31.466 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-361.el9.x86_64 2024-02-14T17:27:31.467 INFO:teuthology.task.kernel:Running kernel on smithi027: 5.14.0-361.el9.x86_64 2024-02-14T17:27:31.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2024-02-14T17:27:33.588 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-02-14T17:27:33.659 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-02-14T17:27:33.695 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-02-14T17:27:35.093 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-02-14T17:27:35.137 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-02-14T17:27:35.179 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-02-14T17:27:36.386 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.0 MB 00:00 2024-02-14T17:27:36.430 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 9.0 MB/s | 8.0 MB 00:00 2024-02-14T17:27:36.650 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.0 MB 00:01 2024-02-14T17:27:39.293 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-02-14T17:27:39.665 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 8.9 MB/s | 19 MB 00:02 2024-02-14T17:27:44.129 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 6.1 MB 00:00 2024-02-14T17:27:44.513 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.1 MB 00:01 2024-02-14T17:27:45.580 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 15 kB 00:00 2024-02-14T17:27:47.369 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 8.8 kB/s | 15 kB 00:01 2024-02-14T17:27:47.631 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2024-02-14T17:27:48.595 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 19 MB/s | 20 MB 00:01 2024-02-14T17:27:52.530 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-02-14T17:27:53.729 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-02-14T17:27:54.820 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-14T17:27:54.820 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-14T17:27:54.910 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-02-14T17:27:54.922 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-02-14T17:27:54.922 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-02-14T17:27:54.922 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-02-14T17:27:54.922 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-02-14T17:27:54.922 INFO:teuthology.orchestra.run.smithi096.stdout: kernel x86_64 5.14.0-420.el9 CentOS-BaseOS 6.0 M 2024-02-14T17:27:54.922 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-02-14T17:27:54.922 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core x86_64 5.14.0-420.el9 CentOS-BaseOS 21 M 2024-02-14T17:27:54.922 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules x86_64 5.14.0-420.el9 CentOS-BaseOS 39 M 2024-02-14T17:27:54.922 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core x86_64 5.14.0-420.el9 CentOS-BaseOS 33 M 2024-02-14T17:27:54.922 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:27:54.922 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-02-14T17:27:54.922 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-02-14T17:27:54.923 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-02-14T17:27:54.923 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:27:54.923 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 99 M 2024-02-14T17:27:54.923 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 122 M 2024-02-14T17:27:54.923 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-02-14T17:27:55.481 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): kernel-5.14.0-420.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-02-14T17:27:56.102 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-14T17:27:56.103 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-14T17:27:56.193 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-02-14T17:27:56.198 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-02-14T17:27:56.199 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-02-14T17:27:56.199 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-02-14T17:27:56.199 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-02-14T17:27:56.199 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-420.el9 CentOS-BaseOS 6.0 M 2024-02-14T17:27:56.199 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-02-14T17:27:56.199 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core x86_64 5.14.0-420.el9 CentOS-BaseOS 21 M 2024-02-14T17:27:56.199 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules x86_64 5.14.0-420.el9 CentOS-BaseOS 39 M 2024-02-14T17:27:56.199 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core x86_64 5.14.0-420.el9 CentOS-BaseOS 33 M 2024-02-14T17:27:56.199 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:27:56.199 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-02-14T17:27:56.199 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-02-14T17:27:56.199 INFO:teuthology.orchestra.run.smithi027.stdout:Install 4 Packages 2024-02-14T17:27:56.199 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:27:56.200 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 99 M 2024-02-14T17:27:56.200 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 122 M 2024-02-14T17:27:56.200 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-02-14T17:27:56.265 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): kernel-core-5.14.0-420.el9.x86_64.rpm 16 MB/s | 21 MB 00:01 2024-02-14T17:27:56.990 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): kernel-modules-core-5.14.0-420.el9.x86_6 22 MB/s | 33 MB 00:01 2024-02-14T17:27:57.147 INFO:teuthology.orchestra.run.smithi027.stdout:(1/4): kernel-5.14.0-420.el9.x86_64.rpm 6.3 MB/s | 6.0 MB 00:00 2024-02-14T17:27:57.622 INFO:teuthology.orchestra.run.smithi027.stdout:(2/4): kernel-core-5.14.0-420.el9.x86_64.rpm 15 MB/s | 21 MB 00:01 2024-02-14T17:27:57.748 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): kernel-modules-5.14.0-420.el9.x86_64.rpm 14 MB/s | 39 MB 00:02 2024-02-14T17:27:57.751 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-02-14T17:27:57.752 INFO:teuthology.orchestra.run.smithi096.stdout:Total 35 MB/s | 99 MB 00:02 2024-02-14T17:27:57.752 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-02-14T17:27:58.122 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-02-14T17:27:58.122 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-02-14T17:27:58.423 INFO:teuthology.orchestra.run.smithi027.stdout:(3/4): kernel-modules-5.14.0-420.el9.x86_64.rpm 18 MB/s | 39 MB 00:02 2024-02-14T17:27:59.162 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-02-14T17:27:59.162 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-02-14T17:27:59.497 INFO:teuthology.orchestra.run.smithi027.stdout:(4/4): kernel-modules-core-5.14.0-420.el9.x86_6 14 MB/s | 33 MB 00:02 2024-02-14T17:27:59.500 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-02-14T17:27:59.501 INFO:teuthology.orchestra.run.smithi027.stdout:Total 30 MB/s | 99 MB 00:03 2024-02-14T17:27:59.501 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-02-14T17:27:59.949 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-02-14T17:27:59.949 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-02-14T17:28:01.009 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-02-14T17:28:01.010 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-02-14T17:28:01.548 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-02-14T17:28:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-core-5.14.0-420.el9.x86_64 1/4 2024-02-14T17:28:02.325 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:28:02.668 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 778 kB/s | 19 MB 00:25 2024-02-14T17:28:03.561 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-02-14T17:28:04.240 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-core-5.14.0-420.el9.x86_64 1/4 2024-02-14T17:28:04.373 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:28:05.083 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:28:05.267 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:28:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:28:07.327 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 6.6 MB/s | 6.1 MB 00:00 2024-02-14T17:28:07.510 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:28:08.987 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2024-02-14T17:28:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:28:10.079 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:28:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:28:12.799 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:28:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:28:17.885 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:28:30.579 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 971 kB/s | 20 MB 00:21 2024-02-14T17:28:35.802 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-02-14T17:28:38.183 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-14T17:28:38.184 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-14T17:28:38.279 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-02-14T17:28:38.294 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-02-14T17:28:38.294 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-02-14T17:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-02-14T17:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-02-14T17:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 5.14.0-420.el9 CentOS-BaseOS 6.0 M 2024-02-14T17:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-02-14T17:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core x86_64 5.14.0-420.el9 CentOS-BaseOS 21 M 2024-02-14T17:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules x86_64 5.14.0-420.el9 CentOS-BaseOS 39 M 2024-02-14T17:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-core x86_64 5.14.0-420.el9 CentOS-BaseOS 33 M 2024-02-14T17:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-02-14T17:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-02-14T17:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-02-14T17:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout:Install 4 Packages 2024-02-14T17:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-02-14T17:28:38.296 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 99 M 2024-02-14T17:28:38.296 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 122 M 2024-02-14T17:28:38.296 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-02-14T17:28:39.112 INFO:teuthology.orchestra.run.smithi040.stdout:(1/4): kernel-5.14.0-420.el9.x86_64.rpm 7.4 MB/s | 6.0 MB 00:00 2024-02-14T17:28:39.653 INFO:teuthology.orchestra.run.smithi040.stdout:(2/4): kernel-core-5.14.0-420.el9.x86_64.rpm 15 MB/s | 21 MB 00:01 2024-02-14T17:28:40.845 INFO:teuthology.orchestra.run.smithi040.stdout:(3/4): kernel-modules-core-5.14.0-420.el9.x86_6 19 MB/s | 33 MB 00:01 2024-02-14T17:28:41.195 INFO:teuthology.orchestra.run.smithi040.stdout:(4/4): kernel-modules-5.14.0-420.el9.x86_64.rpm 13 MB/s | 39 MB 00:02 2024-02-14T17:28:41.198 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-02-14T17:28:41.198 INFO:teuthology.orchestra.run.smithi040.stdout:Total 34 MB/s | 99 MB 00:02 2024-02-14T17:28:41.199 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-02-14T17:28:41.650 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-02-14T17:28:41.650 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-02-14T17:28:42.686 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-02-14T17:28:42.687 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-02-14T17:28:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:28:44.054 INFO:teuthology.orchestra.run.smithi096.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-14T17:28:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:28:44.123 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:28:44.567 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:28:44.567 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-5.14.0-420.el9.x86_64 1/4 2024-02-14T17:28:44.567 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:28:44.567 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:28:44.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:28:44.989 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:28:44.990 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-02-14T17:28:44.990 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-5.14.0-420.el9.x86_64 2024-02-14T17:28:44.990 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core-5.14.0-420.el9.x86_64 2024-02-14T17:28:44.990 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-5.14.0-420.el9.x86_64 2024-02-14T17:28:44.990 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core-5.14.0-420.el9.x86_64 2024-02-14T17:28:44.990 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:28:44.990 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-02-14T17:28:45.216 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-02-14T17:28:45.505 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-02-14T17:28:45.630 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-420.el9.x86_64 2024-02-14T17:28:45.630 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-420.el9.x86_64 2024-02-14T17:28:45.630 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-420.el9.x86_64 2024-02-14T17:28:45.630 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-14T17:28:45.630 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-14T17:28:45.630 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-02-14T17:28:45.631 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi096.front.sepia.ceph.com, path=None, version=distro) 2024-02-14T17:28:45.631 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-02-14T17:28:45.740 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-420.el9.x86_64 2024-02-14T17:28:45.740 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-420.el9.x86_64 2024-02-14T17:28:45.740 DEBUG:teuthology.orchestra.run.smithi096:> sudo rpm -qi grub2-tools 2024-02-14T17:28:45.774 INFO:teuthology.orchestra.run.smithi096.stdout:Name : grub2-tools 2024-02-14T17:28:45.774 INFO:teuthology.orchestra.run.smithi096.stdout:Epoch : 1 2024-02-14T17:28:45.774 INFO:teuthology.orchestra.run.smithi096.stdout:Version : 2.06 2024-02-14T17:28:45.774 INFO:teuthology.orchestra.run.smithi096.stdout:Release : 68.el9 2024-02-14T17:28:45.774 INFO:teuthology.orchestra.run.smithi096.stdout:Architecture: x86_64 2024-02-14T17:28:45.774 INFO:teuthology.orchestra.run.smithi096.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-14T17:28:45.774 INFO:teuthology.orchestra.run.smithi096.stdout:Group : Unspecified 2024-02-14T17:28:45.774 INFO:teuthology.orchestra.run.smithi096.stdout:Size : 8264254 2024-02-14T17:28:45.774 INFO:teuthology.orchestra.run.smithi096.stdout:License : GPLv3+ 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout:Packager : builder@centos.org 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout:Vendor : CentOS 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout:Summary : Support tools for GRUB. 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout:Description : 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-14T17:28:45.775 INFO:teuthology.orchestra.run.smithi096.stdout:hardware devices. 2024-02-14T17:28:45.776 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:28:45.776 INFO:teuthology.orchestra.run.smithi096.stdout:This subpackage provides tools for support of all platforms. 2024-02-14T17:28:45.776 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-14T17:28:45.777 INFO:teuthology.task.kernel:Updating grub on smithi096 to boot 5.14.0-420.el9.x86_64 2024-02-14T17:28:45.777 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-14T17:28:45.904 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-core-5.14.0-420.el9.x86_64 1/4 2024-02-14T17:28:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:28:46.403 INFO:teuthology.orchestra.run.smithi096.stderr:Generating grub configuration file ... 2024-02-14T17:28:47.777 INFO:teuthology.orchestra.run.smithi096.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-14T17:28:47.827 INFO:teuthology.orchestra.run.smithi096.stderr:done 2024-02-14T17:28:47.829 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-02-14T17:28:47.844 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.CtVB1KCht8 2024-02-14T17:28:47.845 DEBUG:teuthology.orchestra.run.smithi096:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.CtVB1KCht8 2024-02-14T17:28:47.913 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.CtVB1KCht8 2024-02-14T17:28:48.057 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.CtVB1KCht8 is 6KB 2024-02-14T17:28:48.105 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.CtVB1KCht8 2024-02-14T17:28:48.119 DEBUG:teuthology.orchestra.run.smithi096:> sudo /bin/ls /boot/loader/entries || true 2024-02-14T17:28:48.184 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-14T17:28:48.184 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-14T17:28:48.184 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-14T17:28:48.184 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-5.14.0-420.el9.x86_64.conf 2024-02-14T17:28:48.185 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-420.el9.x86_64 2024-02-14T17:28:48.337 DEBUG:teuthology.orchestra.run.smithi096:> sudo shutdown -r now 2024-02-14T17:28:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:28:48.692 INFO:teuthology.orchestra.run.smithi027.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-14T17:28:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:28:48.765 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:28:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:28:49.090 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:28:49.163 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:28:49.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-420.el9.x86_64 1/4 2024-02-14T17:28:49.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:28:49.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:28:49.631 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:28:49.631 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:28:49.631 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-02-14T17:28:49.631 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-5.14.0-420.el9.x86_64 2024-02-14T17:28:49.632 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core-5.14.0-420.el9.x86_64 2024-02-14T17:28:49.632 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-5.14.0-420.el9.x86_64 2024-02-14T17:28:49.632 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core-5.14.0-420.el9.x86_64 2024-02-14T17:28:49.632 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:28:49.632 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-02-14T17:28:50.137 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-02-14T17:28:50.284 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-420.el9.x86_64 2024-02-14T17:28:50.285 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-420.el9.x86_64 2024-02-14T17:28:50.285 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-420.el9.x86_64 2024-02-14T17:28:50.285 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-14T17:28:50.285 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-14T17:28:50.285 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-14T17:28:50.285 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=None, version=distro) 2024-02-14T17:28:50.286 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-02-14T17:28:50.398 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-420.el9.x86_64 2024-02-14T17:28:50.398 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-420.el9.x86_64 2024-02-14T17:28:50.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2024-02-14T17:28:50.443 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2024-02-14T17:28:50.443 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2024-02-14T17:28:50.443 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.06 2024-02-14T17:28:50.443 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 68.el9 2024-02-14T17:28:50.443 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2024-02-14T17:28:50.444 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-14T17:28:50.444 INFO:teuthology.orchestra.run.smithi027.stdout:Group : Unspecified 2024-02-14T17:28:50.444 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 8264254 2024-02-14T17:28:50.444 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2024-02-14T17:28:50.444 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-14T17:28:50.444 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-14T17:28:50.444 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-14T17:28:50.444 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-14T17:28:50.444 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : builder@centos.org 2024-02-14T17:28:50.444 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2024-02-14T17:28:50.444 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-14T17:28:50.444 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2024-02-14T17:28:50.444 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2024-02-14T17:28:50.445 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:28:50.445 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-14T17:28:50.445 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-14T17:28:50.445 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-14T17:28:50.445 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2024-02-14T17:28:50.445 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:28:50.445 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2024-02-14T17:28:50.446 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-14T17:28:50.446 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 5.14.0-420.el9.x86_64 2024-02-14T17:28:50.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-14T17:28:50.941 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-02-14T17:28:52.316 INFO:teuthology.orchestra.run.smithi027.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-14T17:28:52.334 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-02-14T17:28:52.336 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-02-14T17:28:52.352 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.9VHrNYPu1d 2024-02-14T17:28:52.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9VHrNYPu1d 2024-02-14T17:28:52.424 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.9VHrNYPu1d 2024-02-14T17:28:52.566 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.9VHrNYPu1d is 6KB 2024-02-14T17:28:52.615 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.9VHrNYPu1d 2024-02-14T17:28:52.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2024-02-14T17:28:52.702 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-14T17:28:52.702 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-14T17:28:52.702 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-14T17:28:52.702 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-420.el9.x86_64.conf 2024-02-14T17:28:52.703 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-420.el9.x86_64 2024-02-14T17:28:52.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2024-02-14T17:28:54.150 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:28:54.261 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:28:59.342 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:29:18.339 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-02-14T17:29:18.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-02-14T17:29:18.340 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:29:22.870 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-14T17:29:22.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-02-14T17:29:22.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:29:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:29:30.210 INFO:teuthology.orchestra.run.smithi040.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-14T17:29:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-02-14T17:29:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:29:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:29:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-5.14.0-420.el9.x86_64 1/4 2024-02-14T17:29:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:29:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:29:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:29:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-02-14T17:29:31.170 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-02-14T17:29:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-5.14.0-420.el9.x86_64 2024-02-14T17:29:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core-5.14.0-420.el9.x86_64 2024-02-14T17:29:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-5.14.0-420.el9.x86_64 2024-02-14T17:29:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-core-5.14.0-420.el9.x86_64 2024-02-14T17:29:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-02-14T17:29:31.170 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-02-14T17:29:31.682 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-02-14T17:29:31.810 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-420.el9.x86_64 2024-02-14T17:29:31.810 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-420.el9.x86_64 2024-02-14T17:29:31.811 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-420.el9.x86_64 2024-02-14T17:29:31.811 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-14T17:29:31.811 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-14T17:29:31.811 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-14T17:29:31.811 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=None, version=distro) 2024-02-14T17:29:31.811 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-02-14T17:29:31.924 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-420.el9.x86_64 2024-02-14T17:29:31.924 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-420.el9.x86_64 2024-02-14T17:29:31.924 DEBUG:teuthology.orchestra.run.smithi040:> sudo rpm -qi grub2-tools 2024-02-14T17:29:31.970 INFO:teuthology.orchestra.run.smithi040.stdout:Name : grub2-tools 2024-02-14T17:29:31.970 INFO:teuthology.orchestra.run.smithi040.stdout:Epoch : 1 2024-02-14T17:29:31.970 INFO:teuthology.orchestra.run.smithi040.stdout:Version : 2.06 2024-02-14T17:29:31.970 INFO:teuthology.orchestra.run.smithi040.stdout:Release : 68.el9 2024-02-14T17:29:31.970 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: x86_64 2024-02-14T17:29:31.970 INFO:teuthology.orchestra.run.smithi040.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-14T17:29:31.970 INFO:teuthology.orchestra.run.smithi040.stdout:Group : Unspecified 2024-02-14T17:29:31.970 INFO:teuthology.orchestra.run.smithi040.stdout:Size : 8264254 2024-02-14T17:29:31.970 INFO:teuthology.orchestra.run.smithi040.stdout:License : GPLv3+ 2024-02-14T17:29:31.970 INFO:teuthology.orchestra.run.smithi040.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-14T17:29:31.970 INFO:teuthology.orchestra.run.smithi040.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout:Packager : builder@centos.org 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout:Vendor : CentOS 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout:Summary : Support tools for GRUB. 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout:Description : 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout:hardware devices. 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-02-14T17:29:31.971 INFO:teuthology.orchestra.run.smithi040.stdout:This subpackage provides tools for support of all platforms. 2024-02-14T17:29:31.972 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-14T17:29:31.972 INFO:teuthology.task.kernel:Updating grub on smithi040 to boot 5.14.0-420.el9.x86_64 2024-02-14T17:29:31.973 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-14T17:29:32.529 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2024-02-14T17:29:33.883 INFO:teuthology.orchestra.run.smithi040.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-14T17:29:33.911 INFO:teuthology.orchestra.run.smithi040.stderr:done 2024-02-14T17:29:33.913 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-02-14T17:29:33.928 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.lfhLtPJxJj 2024-02-14T17:29:33.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lfhLtPJxJj 2024-02-14T17:29:34.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.lfhLtPJxJj 2024-02-14T17:29:34.144 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.lfhLtPJxJj is 6KB 2024-02-14T17:29:34.192 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.lfhLtPJxJj 2024-02-14T17:29:34.207 DEBUG:teuthology.orchestra.run.smithi040:> sudo /bin/ls /boot/loader/entries || true 2024-02-14T17:29:34.276 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-14T17:29:34.276 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-14T17:29:34.276 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-14T17:29:34.276 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-420.el9.x86_64.conf 2024-02-14T17:29:34.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-420.el9.x86_64 2024-02-14T17:29:34.465 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2024-02-14T17:29:36.832 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-02-14T17:29:41.436 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-02-14T17:29:45.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-02-14T17:29:45.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:29:48.892 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-02-14T17:29:50.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-02-14T17:29:50.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:29:53.500 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-02-14T17:30:00.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-02-14T17:30:00.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:30:04.471 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-14T17:30:04.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-02-14T17:30:04.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:30:05.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-02-14T17:30:05.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:30:08.572 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-02-14T17:30:22.912 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-02-14T17:30:23.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-02-14T17:30:23.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:30:31.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-02-14T17:30:31.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:30:32.758 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-02-14T17:30:33.460 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-02-14T17:30:33.460 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-420.el9.x86_64"... 2024-02-14T17:30:33.460 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-02-14T17:30:33.475 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-420.el9.x86_64 2024-02-14T17:30:33.475 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-420.el9.x86_64 vs 5.14.0-420.el9.x86_64 2024-02-14T17:30:33.475 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-14T17:30:33.475 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-14T17:30:34.476 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-02-14T17:30:34.477 DEBUG:teuthology.orchestra.run.smithi096:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-14T17:30:34.540 INFO:teuthology.orchestra.run.smithi096.stdout:ttyS1 2024-02-14T17:30:34.578 DEBUG:teuthology.parallel:result is None 2024-02-14T17:31:23.577 DEBUG:teuthology.orchestra.remote:timed out 2024-02-14T17:31:31.924 DEBUG:teuthology.orchestra.remote:timed out 2024-02-14T17:31:41.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-02-14T17:31:41.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:31:41.865 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-02-14T17:31:42.251 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-02-14T17:31:42.251 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-420.el9.x86_64"... 2024-02-14T17:31:42.251 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-02-14T17:31:42.267 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-420.el9.x86_64 2024-02-14T17:31:42.267 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-420.el9.x86_64 vs 5.14.0-420.el9.x86_64 2024-02-14T17:31:42.267 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-14T17:31:42.267 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-14T17:31:43.269 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-14T17:31:43.269 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-14T17:31:43.363 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2024-02-14T17:31:43.398 DEBUG:teuthology.parallel:result is None 2024-02-14T17:31:43.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-02-14T17:31:43.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:31:44.199 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-02-14T17:31:44.593 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-02-14T17:31:44.593 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-420.el9.x86_64"... 2024-02-14T17:31:44.593 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-02-14T17:31:44.608 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-420.el9.x86_64 2024-02-14T17:31:44.609 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-420.el9.x86_64 vs 5.14.0-420.el9.x86_64 2024-02-14T17:31:44.609 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-14T17:31:44.609 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-14T17:31:45.610 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-14T17:31:45.610 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-14T17:31:45.691 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2024-02-14T17:31:45.736 DEBUG:teuthology.parallel:result is None 2024-02-14T17:31:45.736 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-14T17:31:45.744 INFO:teuthology.task.internal:Creating test directory... 2024-02-14T17:31:45.745 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-14T17:31:45.747 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-14T17:31:45.749 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-14T17:31:45.768 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-14T17:31:45.776 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-14T17:31:45.784 INFO:teuthology.task.internal:Creating archive directory... 2024-02-14T17:31:45.785 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-14T17:31:45.806 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-14T17:31:45.808 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-14T17:31:45.840 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-14T17:31:45.851 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-14T17:31:45.851 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-14T17:31:45.871 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-14T17:31:45.874 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-14T17:31:45.906 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-14T17:31:45.909 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-14T17:31:45.914 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-14T17:31:45.919 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-14T17:31:45.927 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-14T17:31:45.928 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-14T17:31:45.930 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-14T17:31:45.938 INFO:teuthology.task.internal:Configuring sudo... 2024-02-14T17:31:45.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-14T17:31:45.973 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-14T17:31:45.976 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-14T17:31:46.006 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-14T17:31:46.017 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-14T17:31:46.017 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-14T17:31:46.048 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-14T17:31:46.051 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-14T17:31:46.067 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-14T17:31:46.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-14T17:31:46.201 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-14T17:31:46.285 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-14T17:31:46.331 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:31:46.332 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-14T17:31:46.403 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-14T17:31:46.449 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-14T17:31:46.506 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-14T17:31:46.582 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-14T17:31:46.635 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-02-14T17:31:46.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-14T17:31:46.705 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-14T17:31:46.749 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-14T17:31:46.800 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-14T17:31:46.881 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-14T17:31:46.930 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-02-14T17:31:46.930 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-14T17:31:46.999 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-02-14T17:31:47.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-02-14T17:31:47.005 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2024-02-14T17:31:47.047 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-14T17:31:47.055 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-14T17:31:47.092 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-14T17:31:47.585 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-14T17:31:47.592 INFO:teuthology.task.internal:Starting timer... 2024-02-14T17:31:47.592 INFO:teuthology.run_tasks:Running task pcp... 2024-02-14T17:31:47.603 INFO:teuthology.run_tasks:Running task selinux... 2024-02-14T17:31:47.612 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-14T17:31:47.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-02-14T17:31:47.685 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-02-14T17:31:47.687 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2024-02-14T17:31:47.743 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2024-02-14T17:31:47.745 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2024-02-14T17:31:47.829 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2024-02-14T17:31:47.831 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-14T17:31:47.831 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-02-14T17:31:47.856 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-02-14T17:31:47.857 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2024-02-14T17:31:47.883 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2024-02-14T17:31:47.883 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2024-02-14T17:31:47.908 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2024-02-14T17:31:47.909 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi096.front.sepia.ceph.com': 'permissive'} 2024-02-14T17:31:47.909 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-14T17:31:47.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:31:47.943 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-14T17:31:47.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:31:47.973 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-14T17:31:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:31:48.002 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-14T17:31:48.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-02-14T17:31:48.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2024-02-14T17:31:48.091 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2024-02-14T17:31:48.135 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-14T17:31:48.145 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-14T17:31:48.151 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-14T17:31:48.288 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-14T17:31:48.304 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-14T17:31:48.306 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi040.front.sepia.ceph.com,smithi096.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-14T17:36:15.866 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi096.front.sepia.ceph.com')] 2024-02-14T17:36:15.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-02-14T17:36:15.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:36:15.973 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-02-14T17:36:16.055 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-02-14T17:36:16.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-02-14T17:36:16.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:36:16.178 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-02-14T17:36:16.256 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-02-14T17:36:16.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-02-14T17:36:16.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:36:16.370 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-02-14T17:36:16.449 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-02-14T17:36:16.449 INFO:teuthology.run_tasks:Running task clock... 2024-02-14T17:36:16.461 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-14T17:36:16.461 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-14T17:36:16.461 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-14T17:36:16.464 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-14T17:36:16.464 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-14T17:36:16.466 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-14T17:36:16.466 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-14T17:36:16.496 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-14T17:36:16.499 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-14T17:36:16.518 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-14T17:36:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-14T17:36:16.523 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-14T17:36:16.540 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-14T17:36:16.549 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2024-02-14T17:36:16.554 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-02-14T17:36:16.563 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2024-02-14T17:36:16.570 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2024-02-14T17:36:16.571 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-02-14T17:36:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-14T17:36:16.584 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2024-02-14T17:36:16.592 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-14T17:36:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-14T17:36:16.601 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-14T17:36:16.611 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-14T17:36:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-14T17:36:16.652 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-02-14T17:36:16.656 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-14T17:36:16.656 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-02-14T17:36:16.656 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:36:16.656 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:36:16.656 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:36:16.656 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:36:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-02-14T17:36:16.669 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-14T17:36:16.669 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-02-14T17:36:16.670 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:36:16.670 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:36:16.670 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:36:16.670 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:36:16.672 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-02-14T17:36:16.675 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-14T17:36:16.675 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-02-14T17:36:16.675 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:36:16.675 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:36:16.675 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:36:16.675 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:36:16.676 INFO:teuthology.run_tasks:Running task pexec... 2024-02-14T17:36:16.687 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-14T17:36:16.687 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi027.front.sepia.ceph.com 2024-02-14T17:36:16.687 DEBUG:teuthology.orchestra.run.smithi027:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-14T17:36:16.688 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi040.front.sepia.ceph.com 2024-02-14T17:36:16.688 DEBUG:teuthology.orchestra.run.smithi040:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-14T17:36:16.688 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi096.front.sepia.ceph.com 2024-02-14T17:36:16.688 DEBUG:teuthology.orchestra.run.smithi096:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-14T17:36:17.299 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:55 ago on Wed 14 Feb 2024 05:35:22 PM UTC. 2024-02-14T17:36:17.310 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:57 ago on Wed 14 Feb 2024 05:35:20 PM UTC. 2024-02-14T17:36:17.324 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:49 ago on Wed 14 Feb 2024 05:35:28 PM UTC. 2024-02-14T17:36:17.468 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-02-14T17:36:17.469 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-02-14T17:36:17.469 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-02-14T17:36:17.469 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-02-14T17:36:17.470 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-02-14T17:36:17.470 INFO:teuthology.orchestra.run.smithi040.stdout: runc x86_64 4:1.1.12-1.el9 CentOS-AppStream 3.1 M 2024-02-14T17:36:17.470 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-02-14T17:36:17.470 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-02-14T17:36:17.470 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-02-14T17:36:17.470 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-02-14T17:36:17.470 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-02-14T17:36:17.470 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-02-14T17:36:17.471 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 3.1 M 2024-02-14T17:36:17.471 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 9.8 M 2024-02-14T17:36:17.471 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-02-14T17:36:17.487 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-02-14T17:36:17.487 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-02-14T17:36:17.487 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-02-14T17:36:17.487 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-02-14T17:36:17.487 INFO:teuthology.orchestra.run.smithi096.stdout: runc x86_64 4:1.1.12-1.el9 CentOS-AppStream 3.1 M 2024-02-14T17:36:17.487 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:36:17.487 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-02-14T17:36:17.487 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-02-14T17:36:17.487 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-02-14T17:36:17.487 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:36:17.488 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 3.1 M 2024-02-14T17:36:17.488 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 9.8 M 2024-02-14T17:36:17.488 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-02-14T17:36:17.493 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-02-14T17:36:17.501 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-02-14T17:36:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-02-14T17:36:17.501 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-02-14T17:36:17.502 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-02-14T17:36:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: runc x86_64 4:1.1.12-1.el9 CentOS-AppStream 3.1 M 2024-02-14T17:36:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:36:17.502 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-02-14T17:36:17.502 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-02-14T17:36:17.502 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-02-14T17:36:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:36:17.502 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 3.1 M 2024-02-14T17:36:17.502 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 9.8 M 2024-02-14T17:36:17.502 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-02-14T17:36:18.035 INFO:teuthology.orchestra.run.smithi040.stdout:runc-1.1.12-1.el9.x86_64.rpm 5.5 MB/s | 3.1 MB 00:00 2024-02-14T17:36:18.036 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-02-14T17:36:18.036 INFO:teuthology.orchestra.run.smithi040.stdout:Total 5.5 MB/s | 3.1 MB 00:00 2024-02-14T17:36:18.036 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-02-14T17:36:18.042 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-02-14T17:36:18.042 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-02-14T17:36:18.082 INFO:teuthology.orchestra.run.smithi096.stdout:runc-1.1.12-1.el9.x86_64.rpm 5.3 MB/s | 3.1 MB 00:00 2024-02-14T17:36:18.082 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-02-14T17:36:18.083 INFO:teuthology.orchestra.run.smithi096.stdout:Total 5.2 MB/s | 3.1 MB 00:00 2024-02-14T17:36:18.083 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-02-14T17:36:18.088 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-02-14T17:36:18.088 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-02-14T17:36:18.090 INFO:teuthology.orchestra.run.smithi027.stdout:runc-1.1.12-1.el9.x86_64.rpm 5.3 MB/s | 3.1 MB 00:00 2024-02-14T17:36:18.090 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-02-14T17:36:18.091 INFO:teuthology.orchestra.run.smithi027.stdout:Total 5.3 MB/s | 3.1 MB 00:00 2024-02-14T17:36:18.091 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-02-14T17:36:18.097 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-02-14T17:36:18.097 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-02-14T17:36:18.117 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-02-14T17:36:18.117 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-02-14T17:36:18.160 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-02-14T17:36:18.161 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-02-14T17:36:18.173 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-02-14T17:36:18.173 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-02-14T17:36:18.335 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-02-14T17:36:18.374 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-02-14T17:36:18.399 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-02-14T17:36:18.425 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-14T17:36:18.469 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-14T17:36:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-14T17:36:18.974 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-14T17:36:19.060 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-14T17:36:19.070 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-14T17:36:19.262 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-14T17:36:19.262 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-02-14T17:36:19.262 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-02-14T17:36:19.262 INFO:teuthology.orchestra.run.smithi040.stdout: runc-4:1.1.12-1.el9.x86_64 2024-02-14T17:36:19.262 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-02-14T17:36:19.262 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-02-14T17:36:19.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-14T17:36:19.374 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:36:19.374 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-02-14T17:36:19.374 INFO:teuthology.orchestra.run.smithi027.stdout: runc-4:1.1.12-1.el9.x86_64 2024-02-14T17:36:19.374 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:36:19.374 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-02-14T17:36:19.381 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-14T17:36:19.381 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:36:19.381 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-02-14T17:36:19.381 INFO:teuthology.orchestra.run.smithi096.stdout: runc-4:1.1.12-1.el9.x86_64 2024-02-14T17:36:19.381 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:36:19.381 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-02-14T17:36:19.529 DEBUG:teuthology.parallel:result is None 2024-02-14T17:36:19.624 DEBUG:teuthology.parallel:result is None 2024-02-14T17:36:19.636 DEBUG:teuthology.parallel:result is None 2024-02-14T17:36:19.636 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-02-14T17:36:19.648 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-02-14T17:36:19.648 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:36:19.648 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-02-14T17:36:19.664 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-14T17:36:19.664 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-02-14T17:36:19.719 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-14T17:36:19.719 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:36:19.720 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-14T17:36:19.720 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:36:19.720 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:36:19.720 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-02-14 17:36:14.814752662 +0000 2024-02-14T17:36:19.720 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-02-14 17:35:53.042318469 +0000 2024-02-14T17:36:19.720 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-02-14 17:35:53.042318469 +0000 2024-02-14T17:36:19.720 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-02-14 17:35:53.042318469 +0000 2024-02-14T17:36:19.720 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-14T17:36:19.789 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-02-14T17:36:19.789 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-02-14T17:36:19.789 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.00024735 s, 2.1 MB/s 2024-02-14T17:36:19.790 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-14T17:36:19.846 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-02-14T17:36:19.902 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-14T17:36:19.902 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:36:19.902 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-14T17:36:19.903 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:36:19.903 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:36:19.903 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-02-14 17:36:14.814752662 +0000 2024-02-14T17:36:19.903 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-02-14 17:35:53.391309400 +0000 2024-02-14T17:36:19.903 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-02-14 17:35:53.391309400 +0000 2024-02-14T17:36:19.903 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-02-14 17:35:53.391309400 +0000 2024-02-14T17:36:19.903 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-14T17:36:19.972 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-02-14T17:36:19.972 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-02-14T17:36:19.972 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000212148 s, 2.4 MB/s 2024-02-14T17:36:19.974 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-14T17:36:20.029 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-02-14T17:36:20.085 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-14T17:36:20.085 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:36:20.085 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-14T17:36:20.085 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:36:20.085 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:36:20.086 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-02-14 17:36:14.814752662 +0000 2024-02-14T17:36:20.086 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-02-14 17:35:53.754299966 +0000 2024-02-14T17:36:20.086 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-02-14 17:35:53.754299966 +0000 2024-02-14T17:36:20.086 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-02-14 17:35:53.754299966 +0000 2024-02-14T17:36:20.086 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-14T17:36:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-02-14T17:36:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-02-14T17:36:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000201517 s, 2.5 MB/s 2024-02-14T17:36:20.156 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-14T17:36:20.213 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-02-14T17:36:20.272 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-14T17:36:20.273 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:36:20.273 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-14T17:36:20.273 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:36:20.273 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:36:20.273 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-02-14 17:36:14.814752662 +0000 2024-02-14T17:36:20.273 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-02-14 17:35:54.110290715 +0000 2024-02-14T17:36:20.273 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-02-14 17:35:54.110290715 +0000 2024-02-14T17:36:20.273 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-02-14 17:35:54.110290715 +0000 2024-02-14T17:36:20.273 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-14T17:36:20.343 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-02-14T17:36:20.344 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-02-14T17:36:20.344 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000201256 s, 2.5 MB/s 2024-02-14T17:36:20.344 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-14T17:36:20.400 DEBUG:teuthology.orchestra.run.smithi027:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-02-14T17:36:20.563 INFO:teuthology.orchestra.run.smithi027.stdout:loop 2024-02-14T17:36:20.564 INFO:tasks.nvme_loop:Connecting nvme_loop smithi027:/dev/vg_nvme/lv_1... 2024-02-14T17:36:20.565 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-02-14T17:36:20.610 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-02-14T17:36:20.657 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/vg_nvme/lv_11 2024-02-14T17:36:20.701 INFO:tasks.nvme_loop:Connecting nvme_loop smithi027:/dev/vg_nvme/lv_2... 2024-02-14T17:36:20.701 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-02-14T17:36:20.746 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-02-14T17:36:20.795 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/vg_nvme/lv_21 2024-02-14T17:36:20.832 INFO:tasks.nvme_loop:Connecting nvme_loop smithi027:/dev/vg_nvme/lv_3... 2024-02-14T17:36:20.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-02-14T17:36:20.880 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-02-14T17:36:20.929 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/vg_nvme/lv_31 2024-02-14T17:36:20.965 INFO:tasks.nvme_loop:Connecting nvme_loop smithi027:/dev/vg_nvme/lv_4... 2024-02-14T17:36:20.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-02-14T17:36:21.013 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-02-14T17:36:21.063 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/vg_nvme/lv_41 2024-02-14T17:36:21.099 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:36:21.100 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-02-14T17:36:21.118 DEBUG:teuthology.orchestra.run.smithi027:> sudo nvme list 2024-02-14T17:36:21.188 INFO:teuthology.orchestra.run.smithi027.stdout:Node SN Model Namespace Usage Format FW Rev 2024-02-14T17:36:21.188 INFO:teuthology.orchestra.run.smithi027.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-02-14T17:36:21.188 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/nvme0n1 CVFT5331001C400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-02-14T17:36:21.189 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/nvme1n1 311c781b5f37a8107fb3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-14T17:36:21.189 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/nvme2n1 a1926a06b6dc83827cec Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-14T17:36:21.189 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/nvme3n1 c9f31ed2f1cabfc0c095 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-14T17:36:21.189 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/nvme4n1 f844bf9d30a0971eb28b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-14T17:36:21.190 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-14T17:36:21.190 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:36:21.190 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/scratch_devs 2024-02-14T17:36:21.261 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-02-14T17:36:21.261 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-02-14T17:36:21.280 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-14T17:36:21.280 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-02-14T17:36:21.338 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-14T17:36:21.338 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:36:21.338 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-14T17:36:21.338 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:36:21.338 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:36:21.338 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-02-14 17:36:14.820620630 +0000 2024-02-14T17:36:21.338 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-02-14 17:35:46.967008048 +0000 2024-02-14T17:36:21.338 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-02-14 17:35:46.967008048 +0000 2024-02-14T17:36:21.338 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-02-14 17:35:46.967008048 +0000 2024-02-14T17:36:21.339 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-14T17:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-02-14T17:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-02-14T17:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000372289 s, 1.4 MB/s 2024-02-14T17:36:21.410 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-14T17:36:21.467 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-02-14T17:36:21.524 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-14T17:36:21.524 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:36:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-14T17:36:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:36:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:36:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-02-14 17:36:14.820620630 +0000 2024-02-14T17:36:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-02-14 17:35:47.311003264 +0000 2024-02-14T17:36:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-02-14 17:35:47.311003264 +0000 2024-02-14T17:36:21.524 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-02-14 17:35:47.311003264 +0000 2024-02-14T17:36:21.525 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-14T17:36:21.592 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-02-14T17:36:21.592 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-02-14T17:36:21.593 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000176825 s, 2.9 MB/s 2024-02-14T17:36:21.594 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-14T17:36:21.650 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-02-14T17:36:21.707 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-14T17:36:21.707 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:36:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-14T17:36:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:36:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:36:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-02-14 17:36:14.820620630 +0000 2024-02-14T17:36:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-02-14 17:35:47.656998452 +0000 2024-02-14T17:36:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-02-14 17:35:47.656998452 +0000 2024-02-14T17:36:21.708 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-02-14 17:35:47.656998452 +0000 2024-02-14T17:36:21.708 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-14T17:36:21.775 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-02-14T17:36:21.775 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-02-14T17:36:21.775 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000185954 s, 2.8 MB/s 2024-02-14T17:36:21.776 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-14T17:36:21.833 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-02-14T17:36:21.889 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-14T17:36:21.889 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:36:21.889 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-14T17:36:21.890 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:36:21.890 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:36:21.890 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-02-14 17:36:14.820620630 +0000 2024-02-14T17:36:21.890 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-02-14 17:35:48.001993654 +0000 2024-02-14T17:36:21.890 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-02-14 17:35:48.001993654 +0000 2024-02-14T17:36:21.890 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-02-14 17:35:48.001993654 +0000 2024-02-14T17:36:21.890 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-14T17:36:21.957 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-02-14T17:36:21.957 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-02-14T17:36:21.957 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000191335 s, 2.7 MB/s 2024-02-14T17:36:21.958 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-14T17:36:22.015 DEBUG:teuthology.orchestra.run.smithi040:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-02-14T17:36:22.158 INFO:teuthology.orchestra.run.smithi040.stdout:loop 2024-02-14T17:36:22.159 INFO:tasks.nvme_loop:Connecting nvme_loop smithi040:/dev/vg_nvme/lv_1... 2024-02-14T17:36:22.160 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-02-14T17:36:22.198 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-02-14T17:36:22.235 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/vg_nvme/lv_11 2024-02-14T17:36:22.272 INFO:tasks.nvme_loop:Connecting nvme_loop smithi040:/dev/vg_nvme/lv_2... 2024-02-14T17:36:22.272 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-02-14T17:36:22.354 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-02-14T17:36:22.390 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/vg_nvme/lv_21 2024-02-14T17:36:22.418 INFO:tasks.nvme_loop:Connecting nvme_loop smithi040:/dev/vg_nvme/lv_3... 2024-02-14T17:36:22.419 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-02-14T17:36:22.498 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-02-14T17:36:22.536 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/vg_nvme/lv_31 2024-02-14T17:36:22.564 INFO:tasks.nvme_loop:Connecting nvme_loop smithi040:/dev/vg_nvme/lv_4... 2024-02-14T17:36:22.564 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-02-14T17:36:22.640 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-02-14T17:36:22.677 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/vg_nvme/lv_41 2024-02-14T17:36:22.712 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-02-14T17:36:22.712 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-02-14T17:36:22.767 DEBUG:teuthology.orchestra.run.smithi040:> sudo nvme list 2024-02-14T17:36:22.833 INFO:teuthology.orchestra.run.smithi040.stdout:Node SN Model Namespace Usage Format FW Rev 2024-02-14T17:36:22.833 INFO:teuthology.orchestra.run.smithi040.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-02-14T17:36:22.833 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/nvme0n1 CVFT5330009J400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-02-14T17:36:22.833 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/nvme1n1 6b252fd59f256fc4ca34 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-14T17:36:22.833 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/nvme2n1 4a41ee4c958baeeff002 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-14T17:36:22.833 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/nvme3n1 af1c9db0af8f0d34ae41 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-14T17:36:22.833 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/nvme4n1 eb321353cc544bde3c85 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-14T17:36:22.834 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-14T17:36:22.834 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-02-14T17:36:22.834 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/scratch_devs 2024-02-14T17:36:22.900 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-02-14T17:36:22.900 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-02-14T17:36:22.919 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-14T17:36:22.919 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2024-02-14T17:36:22.977 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-14T17:36:22.977 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:36:22.977 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-14T17:36:22.977 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:36:22.977 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:36:22.977 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-02-14 17:36:14.899272822 +0000 2024-02-14T17:36:22.977 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-02-14 17:35:45.057980791 +0000 2024-02-14T17:36:22.977 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-02-14 17:35:45.057980791 +0000 2024-02-14T17:36:22.977 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-02-14 17:35:45.057980791 +0000 2024-02-14T17:36:22.978 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-14T17:36:23.046 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-02-14T17:36:23.046 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-02-14T17:36:23.046 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000215939 s, 2.4 MB/s 2024-02-14T17:36:23.047 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-14T17:36:23.104 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2024-02-14T17:36:23.160 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-14T17:36:23.160 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:36:23.161 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-14T17:36:23.161 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:36:23.161 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:36:23.161 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-02-14 17:36:14.899272822 +0000 2024-02-14T17:36:23.161 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-02-14 17:35:45.415972267 +0000 2024-02-14T17:36:23.161 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-02-14 17:35:45.415972267 +0000 2024-02-14T17:36:23.161 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-02-14 17:35:45.415972267 +0000 2024-02-14T17:36:23.161 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-14T17:36:23.227 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-02-14T17:36:23.227 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-02-14T17:36:23.227 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000193642 s, 2.6 MB/s 2024-02-14T17:36:23.229 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-14T17:36:23.286 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2024-02-14T17:36:23.341 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-14T17:36:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:36:23.342 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-14T17:36:23.342 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:36:23.342 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:36:23.342 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-02-14 17:36:14.899272822 +0000 2024-02-14T17:36:23.342 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-02-14 17:35:45.748964338 +0000 2024-02-14T17:36:23.342 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-02-14 17:35:45.748964338 +0000 2024-02-14T17:36:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-02-14 17:35:45.748964338 +0000 2024-02-14T17:36:23.343 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-14T17:36:23.407 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-02-14T17:36:23.407 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-02-14T17:36:23.407 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000198524 s, 2.6 MB/s 2024-02-14T17:36:23.408 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-14T17:36:23.464 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2024-02-14T17:36:23.519 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-14T17:36:23.520 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:36:23.520 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-14T17:36:23.520 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:36:23.520 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:36:23.520 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-02-14 17:36:14.899272822 +0000 2024-02-14T17:36:23.520 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-02-14 17:35:46.077956504 +0000 2024-02-14T17:36:23.520 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-02-14 17:35:46.077956504 +0000 2024-02-14T17:36:23.520 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-02-14 17:35:46.077956504 +0000 2024-02-14T17:36:23.520 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-14T17:36:23.584 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-02-14T17:36:23.584 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-02-14T17:36:23.584 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000205632 s, 2.5 MB/s 2024-02-14T17:36:23.586 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-14T17:36:23.641 DEBUG:teuthology.orchestra.run.smithi096:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-02-14T17:36:23.776 INFO:teuthology.orchestra.run.smithi096.stdout:loop 2024-02-14T17:36:23.778 INFO:tasks.nvme_loop:Connecting nvme_loop smithi096:/dev/vg_nvme/lv_1... 2024-02-14T17:36:23.778 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-02-14T17:36:23.815 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-02-14T17:36:23.852 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/vg_nvme/lv_11 2024-02-14T17:36:23.881 INFO:tasks.nvme_loop:Connecting nvme_loop smithi096:/dev/vg_nvme/lv_2... 2024-02-14T17:36:23.882 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-02-14T17:36:23.961 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-02-14T17:36:24.001 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/vg_nvme/lv_21 2024-02-14T17:36:24.042 INFO:tasks.nvme_loop:Connecting nvme_loop smithi096:/dev/vg_nvme/lv_3... 2024-02-14T17:36:24.042 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-02-14T17:36:24.122 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-02-14T17:36:24.160 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/vg_nvme/lv_31 2024-02-14T17:36:24.190 INFO:tasks.nvme_loop:Connecting nvme_loop smithi096:/dev/vg_nvme/lv_4... 2024-02-14T17:36:24.190 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-02-14T17:36:24.269 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-02-14T17:36:24.308 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/vg_nvme/lv_41 2024-02-14T17:36:24.349 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-02-14T17:36:24.349 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-02-14T17:36:24.406 DEBUG:teuthology.orchestra.run.smithi096:> sudo nvme list 2024-02-14T17:36:24.472 INFO:teuthology.orchestra.run.smithi096.stdout:Node SN Model Namespace Usage Format FW Rev 2024-02-14T17:36:24.472 INFO:teuthology.orchestra.run.smithi096.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-02-14T17:36:24.472 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/nvme0n1 PHFT620400NH400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-02-14T17:36:24.472 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/nvme1n1 efcfc93cf01bf074f5c1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-14T17:36:24.473 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/nvme2n1 7945d4a6074f397999b4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-14T17:36:24.473 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/nvme3n1 f7819e38379f6a7a6851 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-14T17:36:24.473 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/nvme4n1 fbdc637b106fd2109170 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-14T17:36:24.474 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-14T17:36:24.474 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-02-14T17:36:24.474 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/scratch_devs 2024-02-14T17:36:24.540 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-14T17:36:24.644 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '8727ce939fc044a98f215e29696bb1433f5758d1'} 2024-02-14T17:36:24.645 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:36:24.645 INFO:tasks.cephadm:Cluster fsid is 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:36:24.645 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-14T17:36:24.646 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.27', 'mon.b': '172.21.15.40', 'mon.c': '172.21.15.96'} 2024-02-14T17:36:24.646 INFO:tasks.cephadm:First mon is mon.a on smithi027 2024-02-14T17:36:24.646 INFO:tasks.cephadm:First mgr is y 2024-02-14T17:36:24.646 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-14T17:36:24.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo hostname $(hostname -s) 2024-02-14T17:36:24.684 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname $(hostname -s) 2024-02-14T17:36:24.716 DEBUG:teuthology.orchestra.run.smithi096:> sudo hostname $(hostname -s) 2024-02-14T17:36:24.744 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-02-14T17:36:24.744 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:36:24.884 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '8727ce939fc044a98f215e29696bb1433f5758d1', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76734/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1430-g8727ce93', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1430.g8727ce93'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-guits-testing-2024-02-14-1033/8727ce939fc044a98f215e29696bb1433f5758d1/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-14 13:20:25.883975', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-testing-2024-02-14-1033', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-guits-testing-2024-02-14-1033/8727ce939fc044a98f215e29696bb1433f5758d1/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-02-14T17:36:25.019 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-guits-testing-2024-02-14-1033, sha1 8727ce939fc044a98f215e29696bb1433f5758d1 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:36:25.020 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-guits-testing-2024-02-14-1033/8727ce939fc044a98f215e29696bb1433f5758d1/centos/9/x86_64/flavors/default/cephadm 2024-02-14T17:36:25.020 DEBUG:teuthology.orchestra.run.smithi027:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-guits-testing-2024-02-14-1033/8727ce939fc044a98f215e29696bb1433f5758d1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-14T17:36:25.639 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r--. 1 ubuntu ubuntu 767024 Feb 14 17:36 /home/ubuntu/cephtest/cephadm 2024-02-14T17:36:25.639 DEBUG:teuthology.orchestra.run.smithi040:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-guits-testing-2024-02-14-1033/8727ce939fc044a98f215e29696bb1433f5758d1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-14T17:36:26.273 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r--. 1 ubuntu ubuntu 767024 Feb 14 17:36 /home/ubuntu/cephtest/cephadm 2024-02-14T17:36:26.273 DEBUG:teuthology.orchestra.run.smithi096:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-guits-testing-2024-02-14-1033/8727ce939fc044a98f215e29696bb1433f5758d1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-14T17:36:27.213 INFO:teuthology.orchestra.run.smithi096.stdout:-rw-r--r--. 1 ubuntu ubuntu 767024 Feb 14 17:36 /home/ubuntu/cephtest/cephadm 2024-02-14T17:36:27.214 DEBUG:teuthology.orchestra.run.smithi027:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-14T17:36:27.234 DEBUG:teuthology.orchestra.run.smithi040:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-14T17:36:27.252 DEBUG:teuthology.orchestra.run.smithi096:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-14T17:36:27.282 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 on all hosts... 2024-02-14T17:36:27.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 pull 2024-02-14T17:36:27.285 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 pull 2024-02-14T17:36:27.295 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 pull 2024-02-14T17:36:28.110 INFO:teuthology.orchestra.run.smithi027.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1... 2024-02-14T17:36:28.149 INFO:teuthology.orchestra.run.smithi040.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1... 2024-02-14T17:36:28.149 INFO:teuthology.orchestra.run.smithi096.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1... 2024-02-14T17:36:55.267 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-02-14T17:36:55.267 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)", 2024-02-14T17:36:55.267 INFO:teuthology.orchestra.run.smithi096.stdout: "image_id": "170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86", 2024-02-14T17:36:55.267 INFO:teuthology.orchestra.run.smithi096.stdout: "repo_digests": [ 2024-02-14T17:36:55.267 INFO:teuthology.orchestra.run.smithi096.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b" 2024-02-14T17:36:55.267 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-02-14T17:36:55.267 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-02-14T17:36:55.905 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-02-14T17:36:55.906 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)", 2024-02-14T17:36:55.906 INFO:teuthology.orchestra.run.smithi040.stdout: "image_id": "170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86", 2024-02-14T17:36:55.906 INFO:teuthology.orchestra.run.smithi040.stdout: "repo_digests": [ 2024-02-14T17:36:55.906 INFO:teuthology.orchestra.run.smithi040.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b" 2024-02-14T17:36:55.906 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-02-14T17:36:55.906 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-02-14T17:36:56.269 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-02-14T17:36:56.269 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)", 2024-02-14T17:36:56.269 INFO:teuthology.orchestra.run.smithi027.stdout: "image_id": "170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86", 2024-02-14T17:36:56.269 INFO:teuthology.orchestra.run.smithi027.stdout: "repo_digests": [ 2024-02-14T17:36:56.269 INFO:teuthology.orchestra.run.smithi027.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b" 2024-02-14T17:36:56.269 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-02-14T17:36:56.269 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-02-14T17:36:56.294 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph 2024-02-14T17:36:56.328 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph 2024-02-14T17:36:56.357 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph 2024-02-14T17:36:56.391 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /etc/ceph 2024-02-14T17:36:56.423 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /etc/ceph 2024-02-14T17:36:56.450 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /etc/ceph 2024-02-14T17:36:56.480 INFO:tasks.cephadm:Writing seed config... 2024-02-14T17:36:56.481 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-14T17:36:56.481 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-14T17:36:56.481 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-02-14T17:36:56.481 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-02-14T17:36:56.482 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-14T17:36:56.482 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-14T17:36:56.482 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-14T17:36:56.482 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-14T17:36:56.482 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-14T17:36:56.482 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-14T17:36:56.483 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:36:56.483 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-14T17:36:56.498 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 932ae096-cb5f-11ee-95b9-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-02-14T17:36:56.498 DEBUG:teuthology.orchestra.run.smithi027:mon.a> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.a.service 2024-02-14T17:36:56.540 DEBUG:teuthology.orchestra.run.smithi027:mgr.y> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mgr.y.service 2024-02-14T17:36:56.583 INFO:tasks.cephadm:Bootstrapping... 2024-02-14T17:36:56.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 -v bootstrap --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.27 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:36:56.785 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-02-14T17:36:56.785 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1', '-v', 'bootstrap', '--fsid', '932ae096-cb5f-11ee-95b9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.27', '--skip-admin-label'] 2024-02-14T17:36:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-14T17:36:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-14T17:36:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying podman|docker is present... 2024-02-14T17:36:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-14T17:36:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying lvm2 is present... 2024-02-14T17:36:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying time synchronization is in place... 2024-02-14T17:36:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-14T17:36:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-14T17:36:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-14T17:36:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-02-14T17:36:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-02-14T17:36:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-02-14T17:36:56.861 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-02-14T17:36:56.861 INFO:teuthology.orchestra.run.smithi027.stdout:Repeating the final host check... 2024-02-14T17:36:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-14T17:36:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-14T17:36:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl is present 2024-02-14T17:36:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:lvcreate is present 2024-02-14T17:36:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-14T17:36:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-14T17:36:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-14T17:36:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-02-14T17:36:56.902 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-02-14T17:36:56.908 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-02-14T17:36:56.909 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-02-14T17:36:56.909 INFO:teuthology.orchestra.run.smithi027.stdout:Host looks OK 2024-02-14T17:36:56.909 INFO:teuthology.orchestra.run.smithi027.stdout:Cluster fsid: 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:36:56.909 INFO:teuthology.orchestra.run.smithi027.stdout:Acquiring lock 140625066648624 on /run/cephadm/932ae096-cb5f-11ee-95b9-87774f69a715.lock 2024-02-14T17:36:56.909 INFO:teuthology.orchestra.run.smithi027.stdout:Lock 140625066648624 acquired on /run/cephadm/932ae096-cb5f-11ee-95b9-87774f69a715.lock 2024-02-14T17:36:56.909 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 3300 ... 2024-02-14T17:36:56.910 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 6789 ... 2024-02-14T17:36:56.910 INFO:teuthology.orchestra.run.smithi027.stdout:Base mon IP(s) is [172.21.15.27:3300, 172.21.15.27:6789], mon addrv is [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-02-14T17:36:56.913 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.27 metric 100 2024-02-14T17:36:56.913 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.27 metric 100 2024-02-14T17:36:56.915 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-14T17:36:56.915 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-14T17:36:56.915 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-14T17:36:56.918 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-14T17:36:56.918 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-14T17:36:56.918 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-14T17:36:56.918 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-14T17:36:56.918 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-02-14T17:36:56.918 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-14T17:36:56.919 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-02-14T17:36:56.919 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-02-14T17:36:56.919 INFO:teuthology.orchestra.run.smithi027.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-14T17:36:56.919 INFO:teuthology.orchestra.run.smithi027.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-14T17:36:56.920 INFO:teuthology.orchestra.run.smithi027.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1... 2024-02-14T17:36:57.379 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 2024-02-14T17:36:57.380 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1... 2024-02-14T17:36:57.380 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Getting image source signatures 2024-02-14T17:36:57.380 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:85deddcd78c19bc18770d0f2ed0eee83c11fab2d5e1d2ebb809a77baeec400f6 2024-02-14T17:36:57.380 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:0f642e61a11afb42c0ccb5b05f5809250ad643ed143780d6ff49a5a4c492fc38 2024-02-14T17:36:57.380 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying config sha256:170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 2024-02-14T17:36:57.380 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-14T17:36:58.267 INFO:teuthology.orchestra.run.smithi027.stdout:ceph: stdout ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev) 2024-02-14T17:36:58.267 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph version: ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev) 2024-02-14T17:36:58.268 INFO:teuthology.orchestra.run.smithi027.stdout:Extracting ceph user uid/gid from container image... 2024-02-14T17:36:59.222 INFO:teuthology.orchestra.run.smithi027.stdout:stat: stdout 167 167 2024-02-14T17:36:59.222 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial keys... 2024-02-14T17:37:00.160 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQA7+sxlG/49IxAA1gYOmnaNO/Kxa4wbMUZ8Ww== 2024-02-14T17:37:00.974 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQA8+sxl+tcTHBAARswcOEaGnC90llMlyqic6Q== 2024-02-14T17:37:01.887 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQA9+sxlqRp2EhAAZxmUMNbRXnQaWVyhZei+zA== 2024-02-14T17:37:01.888 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial monmap... 2024-02-14T17:37:02.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-14T17:37:02.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-02-14T17:37:02.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:37:02.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-14T17:37:02.827 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool for a [v2:172.21.15.27:3300,v1:172.21.15.27:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-14T17:37:02.827 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = quincy 2024-02-14T17:37:02.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: set fsid to 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:37:02.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-14T17:37:02.827 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:37:02.828 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mon... 2024-02-14T17:37:03.682 INFO:teuthology.orchestra.run.smithi027.stdout:create mon.a on 2024-02-14T17:37:04.214 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-14T17:37:04.429 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-932ae096-cb5f-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-932ae096-cb5f-11ee-95b9-87774f69a715.target. 2024-02-14T17:37:04.430 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-932ae096-cb5f-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-932ae096-cb5f-11ee-95b9-87774f69a715.target. 2024-02-14T17:37:04.669 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.a 2024-02-14T17:37:04.669 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.a.service: Unit ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.a.service not loaded. 2024-02-14T17:37:04.847 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-932ae096-cb5f-11ee-95b9-87774f69a715.target.wants/ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-932ae096-cb5f-11ee-95b9-87774f69a715@.service. 2024-02-14T17:37:05.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:05 smithi027 podman[25307]: 2024-02-14 17:37:04.999558086 +0000 UTC m=+0.018250299 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:37:05.407 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-14T17:37:05.407 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-02-14T17:37:05.413 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-14T17:37:05.414 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-02-14T17:37:05.414 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-02-14T17:37:05.414 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-02-14T17:37:05.414 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon to start... 2024-02-14T17:37:05.414 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon... 2024-02-14T17:37:05.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:05 smithi027 podman[25307]: 2024-02-14 17:37:05.292742286 +0000 UTC m=+0.311434491 container init 501d6a2251d8d78adcbb99f2839cab5855a90cd227040d5a6b6b1a039378bdaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-14T17:37:05.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:05 smithi027 podman[25307]: 2024-02-14 17:37:05.298400851 +0000 UTC m=+0.317093056 container start 501d6a2251d8d78adcbb99f2839cab5855a90cd227040d5a6b6b1a039378bdaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240214, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:37:05.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:05 smithi027 ceph-mon[25345]: mkfs 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:37:05.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:05 smithi027 ceph-mon[25345]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-14T17:37:05.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:05 smithi027 bash[25307]: 501d6a2251d8d78adcbb99f2839cab5855a90cd227040d5a6b6b1a039378bdaa 2024-02-14T17:37:05.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:05 smithi027 systemd[1]: Started Ceph mon.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:37:06.474 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:06 smithi027 ceph-mon[25345]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-14T17:37:06.474 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:06 smithi027 ceph-mon[25345]: monmap e1: 1 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:37:06.474 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:06 smithi027 ceph-mon[25345]: fsmap 2024-02-14T17:37:06.474 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:06 smithi027 ceph-mon[25345]: osdmap e1: 0 total, 0 up, 0 in 2024-02-14T17:37:06.475 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:06 smithi027 ceph-mon[25345]: mgrmap e1: no daemons active 2024-02-14T17:37:06.475 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:06 smithi027 ceph-mon[25345]: from='client.? 172.21.15.27:0/3954972954' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:37:06.478 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout cluster: 2024-02-14T17:37:06.478 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout id: 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:37:06.478 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-14T17:37:06.478 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-02-14T17:37:06.478 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout services: 2024-02-14T17:37:06.478 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.541267s) 2024-02-14T17:37:06.479 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-14T17:37:06.479 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-14T17:37:06.479 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-02-14T17:37:06.479 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout data: 2024-02-14T17:37:06.479 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-14T17:37:06.479 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-14T17:37:06.479 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-14T17:37:06.479 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pgs: 2024-02-14T17:37:06.479 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-02-14T17:37:06.479 INFO:teuthology.orchestra.run.smithi027.stdout:mon is available 2024-02-14T17:37:06.479 INFO:teuthology.orchestra.run.smithi027.stdout:Assimilating anything we can from ceph.conf... 2024-02-14T17:37:07.451 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-02-14T17:37:07.451 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-14T17:37:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-14T17:37:07.453 INFO:teuthology.orchestra.run.smithi027.stdout:Generating new minimal ceph.conf... 2024-02-14T17:37:07.547 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:07 smithi027 ceph-mon[25345]: from='client.? 172.21.15.27:0/646306959' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-14T17:37:07.548 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:07 smithi027 ceph-mon[25345]: from='client.? 172.21.15.27:0/646306959' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-14T17:37:08.422 INFO:teuthology.orchestra.run.smithi027.stdout:Restarting the monitor... 2024-02-14T17:37:08.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:08 smithi027 ceph-mon[25345]: from='client.? 172.21.15.27:0/460478253' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:37:08.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:08 smithi027 systemd[1]: Stopping Ceph mon.a for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:37:08.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:08 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a[25322]: 2024-02-14T17:37:08.690+0000 7f088d59d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:37:08.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:08 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a[25322]: 2024-02-14T17:37:08.690+0000 7f088d59d700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-14T17:37:08.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:08 smithi027 podman[25674]: 2024-02-14 17:37:08.884846956 +0000 UTC m=+0.286283262 container died 501d6a2251d8d78adcbb99f2839cab5855a90cd227040d5a6b6b1a039378bdaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-02-14T17:37:11.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:11 smithi027 podman[25674]: 2024-02-14 17:37:11.702969564 +0000 UTC m=+3.104405880 container cleanup 501d6a2251d8d78adcbb99f2839cab5855a90cd227040d5a6b6b1a039378bdaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:37:11.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:11 smithi027 bash[25674]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a 2024-02-14T17:37:13.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:13 smithi027 podman[25694]: 2024-02-14 17:37:13.610333083 +0000 UTC m=+4.728902321 container remove 501d6a2251d8d78adcbb99f2839cab5855a90cd227040d5a6b6b1a039378bdaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214) 2024-02-14T17:37:13.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:13 smithi027 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-14T17:37:13.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:13 smithi027 systemd[1]: Stopped Ceph mon.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:37:13.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:13 smithi027 systemd[1]: Starting Ceph mon.a for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:37:14.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:14 smithi027 podman[25790]: 2024-02-14 17:37:13.951544688 +0000 UTC m=+0.017913150 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:37:15.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:14 smithi027 podman[25790]: 2024-02-14 17:37:14.95301658 +0000 UTC m=+1.019385039 container create 6714e9fe1fa6c0df92ee908c354f6a2e47bd5850947446ec5a6ec2ad7bae8a0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:37:17.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:16 smithi027 podman[25790]: 2024-02-14 17:37:16.998350124 +0000 UTC m=+3.064718586 container init 6714e9fe1fa6c0df92ee908c354f6a2e47bd5850947446ec5a6ec2ad7bae8a0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0) 2024-02-14T17:37:17.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 podman[25790]: 2024-02-14 17:37:17.005072622 +0000 UTC m=+3.071441082 container start 6714e9fe1fa6c0df92ee908c354f6a2e47bd5850947446ec5a6ec2ad7bae8a0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-14T17:37:17.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: set uid:gid to 167:167 (ceph:ceph) 2024-02-14T17:37:17.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev), process ceph-mon, pid 7 2024-02-14T17:37:17.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: pidfile_write: ignore empty --pid-file 2024-02-14T17:37:17.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: load: jerasure load: lrc 2024-02-14T17:37:17.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: RocksDB version: 7.9.2 2024-02-14T17:37:17.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Git sha 0 2024-02-14T17:37:17.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Compile date 2024-02-14 11:59:27 2024-02-14T17:37:17.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: DB SUMMARY 2024-02-14T17:37:17.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: DB Session ID: ADCK0W506QX11C6O0U4D 2024-02-14T17:37:17.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: CURRENT file: CURRENT 2024-02-14T17:37:17.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: IDENTITY file: IDENTITY 2024-02-14T17:37:17.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-14T17:37:17.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-14T17:37:17.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84615 ; 2024-02-14T17:37:17.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.error_if_exists: 0 2024-02-14T17:37:17.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.create_if_missing: 0 2024-02-14T17:37:17.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.paranoid_checks: 1 2024-02-14T17:37:17.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-14T17:37:17.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-14T17:37:17.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-14T17:37:17.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.env: 0x55569303dae0 2024-02-14T17:37:17.487 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.fs: PosixFileSystem 2024-02-14T17:37:17.487 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.info_log: 0x555693b118c0 2024-02-14T17:37:17.487 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_file_opening_threads: 16 2024-02-14T17:37:17.487 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.statistics: (nil) 2024-02-14T17:37:17.487 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.use_fsync: 0 2024-02-14T17:37:17.487 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_log_file_size: 0 2024-02-14T17:37:17.487 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-14T17:37:17.487 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-14T17:37:17.487 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.keep_log_file_num: 1000 2024-02-14T17:37:17.487 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.recycle_log_file_num: 0 2024-02-14T17:37:17.487 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.allow_fallocate: 1 2024-02-14T17:37:17.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.allow_mmap_reads: 0 2024-02-14T17:37:17.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.allow_mmap_writes: 0 2024-02-14T17:37:17.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.use_direct_reads: 0 2024-02-14T17:37:17.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-14T17:37:17.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.create_missing_column_families: 0 2024-02-14T17:37:17.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.db_log_dir: 2024-02-14T17:37:17.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.wal_dir: 2024-02-14T17:37:17.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-14T17:37:17.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-14T17:37:17.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-14T17:37:17.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-14T17:37:17.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-14T17:37:17.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-14T17:37:17.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.advise_random_on_open: 1 2024-02-14T17:37:17.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.db_write_buffer_size: 0 2024-02-14T17:37:17.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.write_buffer_manager: 0x555693ae5400 2024-02-14T17:37:17.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-14T17:37:17.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-14T17:37:17.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-14T17:37:17.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.rate_limiter: (nil) 2024-02-14T17:37:17.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-14T17:37:17.490 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.wal_recovery_mode: 2 2024-02-14T17:37:17.490 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.enable_thread_tracking: 0 2024-02-14T17:37:17.490 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.enable_pipelined_write: 0 2024-02-14T17:37:17.490 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.unordered_write: 0 2024-02-14T17:37:17.490 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-14T17:37:17.490 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-14T17:37:17.490 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-14T17:37:17.490 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-14T17:37:17.490 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.row_cache: None 2024-02-14T17:37:17.491 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.wal_filter: None 2024-02-14T17:37:17.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-14T17:37:17.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.allow_ingest_behind: 0 2024-02-14T17:37:17.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.two_write_queues: 0 2024-02-14T17:37:17.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.manual_wal_flush: 0 2024-02-14T17:37:17.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.wal_compression: 0 2024-02-14T17:37:17.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.atomic_flush: 0 2024-02-14T17:37:17.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-14T17:37:17.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-14T17:37:17.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-14T17:37:17.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.log_readahead_size: 0 2024-02-14T17:37:17.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-14T17:37:17.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.best_efforts_recovery: 0 2024-02-14T17:37:17.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-14T17:37:17.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-14T17:37:17.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.allow_data_in_errors: 0 2024-02-14T17:37:17.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.db_host_id: __hostname__ 2024-02-14T17:37:17.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-14T17:37:17.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_background_jobs: 2 2024-02-14T17:37:17.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_background_compactions: -1 2024-02-14T17:37:17.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_subcompactions: 1 2024-02-14T17:37:17.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-14T17:37:17.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-14T17:37:17.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-14T17:37:17.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_total_wal_size: 0 2024-02-14T17:37:17.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-14T17:37:17.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-14T17:37:17.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-14T17:37:17.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-14T17:37:17.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_open_files: -1 2024-02-14T17:37:17.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.bytes_per_sync: 0 2024-02-14T17:37:17.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-14T17:37:17.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-14T17:37:17.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compaction_readahead_size: 0 2024-02-14T17:37:17.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_background_flushes: -1 2024-02-14T17:37:17.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Compression algorithms supported: 2024-02-14T17:37:17.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-14T17:37:17.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: kZSTD supported: 0 2024-02-14T17:37:17.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: kXpressCompression supported: 0 2024-02-14T17:37:17.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: kLZ4HCCompression supported: 1 2024-02-14T17:37:17.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: kZlibCompression supported: 1 2024-02-14T17:37:17.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: kSnappyCompression supported: 1 2024-02-14T17:37:17.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: kLZ4Compression supported: 1 2024-02-14T17:37:17.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: kBZip2Compression supported: 0 2024-02-14T17:37:17.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-14T17:37:17.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-14T17:37:17.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-02-14T17:37:17.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-14T17:37:17.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-14T17:37:17.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.merge_operator: 2024-02-14T17:37:17.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compaction_filter: None 2024-02-14T17:37:17.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compaction_filter_factory: None 2024-02-14T17:37:17.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.sst_partitioner_factory: None 2024-02-14T17:37:17.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-14T17:37:17.497 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-14T17:37:17.497 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555693b119c0) 2024-02-14T17:37:17.497 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks: 1 2024-02-14T17:37:17.497 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-14T17:37:17.497 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-14T17:37:17.497 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_top_level_index_and_filter: 1 2024-02-14T17:37:17.497 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_type: 0 2024-02-14T17:37:17.497 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_index_type: 0 2024-02-14T17:37:17.497 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_shortening: 1 2024-02-14T17:37:17.497 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-14T17:37:17.497 INFO:journalctl@ceph.mon.a.smithi027.stdout: checksum: 4 2024-02-14T17:37:17.497 INFO:journalctl@ceph.mon.a.smithi027.stdout: no_block_cache: 0 2024-02-14T17:37:17.497 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache: 0x555693b15090 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_name: BinnedLRUCache 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_options: 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: capacity : 536870912 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_shard_bits : 4 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: strict_capacity_limit : 0 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: high_pri_pool_ratio: 0.000 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_compressed: (nil) 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: persistent_cache: (nil) 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size: 4096 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size_deviation: 10 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_restart_interval: 16 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_block_restart_interval: 1 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: metadata_block_size: 4096 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: partition_filters: 0 2024-02-14T17:37:17.498 INFO:journalctl@ceph.mon.a.smithi027.stdout: use_delta_encoding: 1 2024-02-14T17:37:17.499 INFO:journalctl@ceph.mon.a.smithi027.stdout: filter_policy: bloomfilter 2024-02-14T17:37:17.499 INFO:journalctl@ceph.mon.a.smithi027.stdout: whole_key_filtering: 1 2024-02-14T17:37:17.499 INFO:journalctl@ceph.mon.a.smithi027.stdout: verify_compression: 0 2024-02-14T17:37:17.499 INFO:journalctl@ceph.mon.a.smithi027.stdout: read_amp_bytes_per_bit: 0 2024-02-14T17:37:17.499 INFO:journalctl@ceph.mon.a.smithi027.stdout: format_version: 5 2024-02-14T17:37:17.499 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_index_compression: 1 2024-02-14T17:37:17.499 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_align: 0 2024-02-14T17:37:17.499 INFO:journalctl@ceph.mon.a.smithi027.stdout: max_auto_readahead_size: 262144 2024-02-14T17:37:17.499 INFO:journalctl@ceph.mon.a.smithi027.stdout: prepopulate_block_cache: 0 2024-02-14T17:37:17.499 INFO:journalctl@ceph.mon.a.smithi027.stdout: initial_auto_readahead_size: 8192 2024-02-14T17:37:17.499 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_file_reads_for_auto_readahead: 2 2024-02-14T17:37:17.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.write_buffer_size: 33554432 2024-02-14T17:37:17.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_write_buffer_number: 2 2024-02-14T17:37:17.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compression: NoCompression 2024-02-14T17:37:17.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.bottommost_compression: Disabled 2024-02-14T17:37:17.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.prefix_extractor: nullptr 2024-02-14T17:37:17.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-14T17:37:17.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.num_levels: 7 2024-02-14T17:37:17.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-14T17:37:17.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-14T17:37:17.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-14T17:37:17.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-14T17:37:17.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-14T17:37:17.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-14T17:37:17.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-14T17:37:17.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:37:17.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-14T17:37:17.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-14T17:37:17.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-14T17:37:17.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-14T17:37:17.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-14T17:37:17.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compression_opts.level: 32767 2024-02-14T17:37:17.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compression_opts.strategy: 0 2024-02-14T17:37:17.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-14T17:37:17.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:37:17.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compression_opts.enabled: false 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.target_file_size_base: 67108864 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-14T17:37:17.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-14T17:37:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-14T17:37:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-14T17:37:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-14T17:37:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-14T17:37:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-14T17:37:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-14T17:37:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-14T17:37:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.arena_block_size: 1048576 2024-02-14T17:37:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-14T17:37:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-14T17:37:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.disable_auto_compactions: 0 2024-02-14T17:37:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-14T17:37:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-14T17:37:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-14T17:37:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-14T17:37:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-14T17:37:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-14T17:37:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-14T17:37:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-14T17:37:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-14T17:37:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-14T17:37:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.table_properties_collectors: 2024-02-14T17:37:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.inplace_update_support: 0 2024-02-14T17:37:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.bloom_locality: 0 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.max_successive_merges: 0 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.paranoid_file_checks: 0 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.force_consistency_checks: 1 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.report_bg_io_stats: 0 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.ttl: 2592000 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-14T17:37:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.enable_blob_files: false 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.min_blob_size: 0 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.blob_file_size: 268435456 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.blob_file_starting_level: 0 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c3cad39a-e5ea-4313-9601-02743600ce3a 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932237039783, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-14T17:37:17.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-14T17:37:17.507 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932237040835, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81240, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79427, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73736, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707932237, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c3cad39a-e5ea-4313-9601-02743600ce3a", "db_session_id": "ADCK0W506QX11C6O0U4D", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-14T17:37:17.507 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932237040907, "job": 1, "event": "recovery_finished"} 2024-02-14T17:37:17.507 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-14T17:37:17.507 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-14T17:37:17.507 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555693c06000 2024-02-14T17:37:17.507 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: DB pointer 0x555693bf0000 2024-02-14T17:37:17.507 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-14T17:37:17.507 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-14T17:37:17.507 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** DB Stats ** 2024-02-14T17:37:17.507 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:37:17.507 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-14T17:37:17.508 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:37:17.508 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:37:17.508 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-14T17:37:17.508 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:37:17.508 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:37:17.508 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-02-14T17:37:17.508 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-02-14T17:37:17.508 INFO:journalctl@ceph.mon.a.smithi027.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-14T17:37:17.508 INFO:journalctl@ceph.mon.a.smithi027.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-14T17:37:17.508 INFO:journalctl@ceph.mon.a.smithi027.stdout: L0 2/0 81.17 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 115.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-14T17:37:17.508 INFO:journalctl@ceph.mon.a.smithi027.stdout: Sum 2/0 81.17 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 115.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-14T17:37:17.508 INFO:journalctl@ceph.mon.a.smithi027.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 115.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 115.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-14T17:37:17.509 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative compaction: 0.00 GB write, 13.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval compaction: 0.00 GB write, 13.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache BinnedLRUCache@0x555693b15090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: starting mon.a rank 0 at public addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] at bind addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: mon.a@-1(???) e1 preinit fsid 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: mon.a@-1(???).mds e1 new map 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: mon.a@-1(???).mds e1 print_map 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout: e1 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-14T17:37:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-14T17:37:17.511 INFO:journalctl@ceph.mon.a.smithi027.stdout: legacy client fscid: -1 2024-02-14T17:37:17.511 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-02-14T17:37:17.511 INFO:journalctl@ceph.mon.a.smithi027.stdout: No filesystems configured 2024-02-14T17:37:17.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-14T17:37:17.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:37:17.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:37:17.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:37:17.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-14T17:37:17.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-14T17:37:17.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: monmap e1: 1 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:37:17.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: fsmap 2024-02-14T17:37:17.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: osdmap e1: 0 total, 0 up, 0 in 2024-02-14T17:37:17.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 ceph-mon[25828]: mgrmap e1: no daemons active 2024-02-14T17:37:17.854 INFO:teuthology.orchestra.run.smithi027.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-14T17:37:18.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 bash[25790]: 6714e9fe1fa6c0df92ee908c354f6a2e47bd5850947446ec5a6ec2ad7bae8a0c 2024-02-14T17:37:18.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:17 smithi027 systemd[1]: Started Ceph mon.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:37:21.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:21 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/959301857' entity='client.admin' 2024-02-14T17:37:22.787 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-14T17:37:22.817 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:37:22.817 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mgr... 2024-02-14T17:37:22.818 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-14T17:37:22.818 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-14T17:37:23.077 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mgr.y 2024-02-14T17:37:23.077 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mgr.y.service: Unit ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mgr.y.service not loaded. 2024-02-14T17:37:23.237 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-932ae096-cb5f-11ee-95b9-87774f69a715.target.wants/ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-932ae096-cb5f-11ee-95b9-87774f69a715@.service. 2024-02-14T17:37:23.589 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:23 smithi027 podman[26072]: 2024-02-14 17:37:23.476071515 +0000 UTC m=+0.117605231 container create 87bd103ac732c669a341530c99f54500b91e58fdf86c343e02822a0ce65e7cd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=) 2024-02-14T17:37:23.762 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-14T17:37:23.762 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-02-14T17:37:23.770 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-14T17:37:23.771 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-02-14T17:37:23.771 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-02-14T17:37:23.771 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-02-14T17:37:23.781 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-14T17:37:23.781 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-02-14T17:37:23.790 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-14T17:37:23.790 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-02-14T17:37:23.790 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-02-14T17:37:23.791 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-02-14T17:37:23.791 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr to start... 2024-02-14T17:37:23.791 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr... 2024-02-14T17:37:23.930 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:23 smithi027 podman[26072]: 2024-02-14 17:37:23.67009721 +0000 UTC m=+0.311630921 container init 87bd103ac732c669a341530c99f54500b91e58fdf86c343e02822a0ce65e7cd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-14T17:37:23.930 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:23 smithi027 podman[26072]: 2024-02-14 17:37:23.675904807 +0000 UTC m=+0.317438518 container start 87bd103ac732c669a341530c99f54500b91e58fdf86c343e02822a0ce65e7cd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:37:23.930 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:23 smithi027 bash[26072]: 87bd103ac732c669a341530c99f54500b91e58fdf86c343e02822a0ce65e7cd6 2024-02-14T17:37:23.931 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:23 smithi027 systemd[1]: Started Ceph mgr.y for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:37:24.553 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:24 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:24.265+0000 7fa9d095e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:37:24.553 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:24 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:24.363+0000 7fa9d095e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:37:24.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:24 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/3340763837' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-14T17:37:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-02-14T17:37:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-02-14T17:37:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "932ae096-cb5f-11ee-95b9-87774f69a715", 2024-02-14T17:37:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-02-14T17:37:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-14T17:37:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-14T17:37:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-14T17:37:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-14T17:37:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-14T17:37:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-14T17:37:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-14T17:37:25.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-14T17:37:25.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-14T17:37:25.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-14T17:37:25.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-14T17:37:25.080 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:25.080 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-14T17:37:25.081 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-02-14T17:37:05.364027+0000", 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-14T17:37:25.082 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:25.083 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-14T17:37:25.083 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-02-14T17:37:25.083 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (1/15)... 2024-02-14T17:37:25.146 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:25.047+0000 7fa9d095e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:37:25.147 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:25.145+0000 7fa9d095e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:37:25.478 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:25.292+0000 7fa9d095e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:37:25.478 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:25.391+0000 7fa9d095e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:37:25.478 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:25.476+0000 7fa9d095e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:37:26.051 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:25.749+0000 7fa9d095e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:37:26.051 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:25.863+0000 7fa9d095e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:37:26.051 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:25.952+0000 7fa9d095e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:37:26.051 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:26 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:26.049+0000 7fa9d095e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:37:26.396 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:26 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:26.233+0000 7fa9d095e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:37:27.469 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:27 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:27.328+0000 7fa9d095e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:37:27.470 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:27 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:27.468+0000 7fa9d095e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:37:28.117 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:27 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/993441578' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-14T17:37:28.117 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:27 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:27.816+0000 7fa9d095e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:37:28.117 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:27 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:27.913+0000 7fa9d095e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:37:28.117 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:28 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:28.115+0000 7fa9d095e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:37:28.410 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:28 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:28.307+0000 7fa9d095e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:37:28.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-02-14T17:37:28.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-02-14T17:37:28.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "932ae096-cb5f-11ee-95b9-87774f69a715", 2024-02-14T17:37:28.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-02-14T17:37:28.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-14T17:37:28.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-14T17:37:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-14T17:37:28.417 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:28.417 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-14T17:37:28.417 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:28.417 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-14T17:37:28.417 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-14T17:37:28.417 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-14T17:37:28.417 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-14T17:37:28.417 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-14T17:37:28.417 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-14T17:37:28.420 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-14T17:37:28.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-02-14T17:37:05.364027+0000", 2024-02-14T17:37:28.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-14T17:37:28.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:28.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-14T17:37:28.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-02-14T17:37:28.423 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (2/15)... 2024-02-14T17:37:28.733 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:28 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:28.601+0000 7fa9d095e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:37:29.233 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:28 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:28.868+0000 7fa9d095e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:37:29.233 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:29 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:29.111+0000 7fa9d095e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:37:30.233 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:29 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:29.943+0000 7fa9d095e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:37:30.804 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:30.544+0000 7fa9d095e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:37:30.804 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:30.648+0000 7fa9d095e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:37:31.233 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:30.878+0000 7fa9d095e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:37:31.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-mon[25828]: Activating manager daemon y 2024-02-14T17:37:31.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-mon[25828]: mgrmap e2: y(active, starting, since 0.0031135s) 2024-02-14T17:37:31.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-mon[25828]: from='mgr.14100 172.21.15.27:0/3494437197' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:37:31.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-mon[25828]: from='mgr.14100 172.21.15.27:0/3494437197' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:37:31.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-mon[25828]: from='mgr.14100 172.21.15.27:0/3494437197' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:37:31.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-mon[25828]: from='mgr.14100 172.21.15.27:0/3494437197' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:37:31.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-mon[25828]: from='mgr.14100 172.21.15.27:0/3494437197' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:37:31.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-mon[25828]: Manager daemon y is now available 2024-02-14T17:37:31.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-mon[25828]: from='mgr.14100 172.21.15.27:0/3494437197' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:37:31.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-mon[25828]: from='mgr.14100 172.21.15.27:0/3494437197' entity='mgr.y' 2024-02-14T17:37:31.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-mon[25828]: from='mgr.14100 172.21.15.27:0/3494437197' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:37:31.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-mon[25828]: from='mgr.14100 172.21.15.27:0/3494437197' entity='mgr.y' 2024-02-14T17:37:31.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:30 smithi027 ceph-mon[25828]: from='mgr.14100 172.21.15.27:0/3494437197' entity='mgr.y' 2024-02-14T17:37:31.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-02-14T17:37:31.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-02-14T17:37:31.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "932ae096-cb5f-11ee-95b9-87774f69a715", 2024-02-14T17:37:31.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-02-14T17:37:31.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-14T17:37:31.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-14T17:37:31.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-14T17:37:31.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:31.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-14T17:37:31.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-14T17:37:31.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-14T17:37:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-14T17:37:31.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-14T17:37:31.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-14T17:37:31.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-14T17:37:31.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:31.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-14T17:37:31.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-14T17:37:31.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-14T17:37:31.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-14T17:37:31.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-14T17:37:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-02-14T17:37:05.364027+0000", 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-02-14T17:37:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (3/15)... 2024-02-14T17:37:32.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:31 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/491012095' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-14T17:37:32.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:31 smithi027 ceph-mon[25828]: mgrmap e3: y(active, since 1.00766s) 2024-02-14T17:37:34.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:33 smithi027 ceph-mon[25828]: mgrmap e4: y(active, since 2s) 2024-02-14T17:37:34.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:34 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/2390854398' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-14T17:37:35.044 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-02-14T17:37:35.044 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-02-14T17:37:35.044 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "932ae096-cb5f-11ee-95b9-87774f69a715", 2024-02-14T17:37:35.044 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-02-14T17:37:35.044 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-14T17:37:35.044 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-14T17:37:35.044 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-14T17:37:35.044 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:35.044 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-14T17:37:35.044 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-14T17:37:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-14T17:37:35.046 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-14T17:37:35.046 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-14T17:37:35.046 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-14T17:37:35.049 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:35.049 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-14T17:37:35.049 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-14T17:37:35.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-14T17:37:35.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-14T17:37:35.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-14T17:37:35.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-14T17:37:35.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-14T17:37:35.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-14T17:37:35.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-14T17:37:35.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:35.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:35.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-14T17:37:35.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-14T17:37:35.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-02-14T17:37:05.364027+0000", 2024-02-14T17:37:35.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-14T17:37:35.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-02-14T17:37:35.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-14T17:37:35.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-02-14T17:37:35.052 INFO:teuthology.orchestra.run.smithi027.stdout:mgr is available 2024-02-14T17:37:36.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:35 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/99486414' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-14T17:37:36.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:35 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/99486414' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-14T17:37:36.231 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-02-14T17:37:36.232 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-02-14T17:37:36.232 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:37:36.232 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-14T17:37:36.232 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-14T17:37:36.232 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-14T17:37:36.232 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-14T17:37:36.232 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-02-14T17:37:36.232 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-14T17:37:36.232 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-14T17:37:36.232 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-02-14T17:37:36.232 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-02-14T17:37:36.232 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-14T17:37:36.232 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-14T17:37:36.233 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling cephadm module... 2024-02-14T17:37:37.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:37 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/177036987' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-14T17:37:37.233 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:37 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: ignoring --setuser ceph since I am not root 2024-02-14T17:37:37.233 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:37 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: ignoring --setgroup ceph since I am not root 2024-02-14T17:37:37.617 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:37 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:37.616+0000 7ff6e01c5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:37:37.880 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:37 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:37.714+0000 7ff6e01c5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:37:38.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:38 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/177036987' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-14T17:37:38.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:38 smithi027 ceph-mon[25828]: mgrmap e5: y(active, since 6s) 2024-02-14T17:37:38.548 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:38 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:38.394+0000 7ff6e01c5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:37:38.548 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:38 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:38.494+0000 7ff6e01c5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:37:38.829 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:38 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:38.642+0000 7ff6e01c5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:37:38.829 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:38 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:38.741+0000 7ff6e01c5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:37:38.829 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:38 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:38.828+0000 7ff6e01c5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:37:38.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-02-14T17:37:38.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-14T17:37:38.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-14T17:37:38.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-14T17:37:38.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-14T17:37:38.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-02-14T17:37:38.852 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-02-14T17:37:38.852 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 5... 2024-02-14T17:37:39.085 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:39 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/4044357556' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-14T17:37:39.351 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:39 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:39.119+0000 7ff6e01c5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:37:39.351 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:39 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:39.233+0000 7ff6e01c5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:37:39.351 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:39 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:39.325+0000 7ff6e01c5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:37:39.608 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:39 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:39.424+0000 7ff6e01c5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:37:39.608 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:39 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:39.606+0000 7ff6e01c5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:37:40.983 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:40 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:40.618+0000 7ff6e01c5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:37:40.983 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:40 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:40.754+0000 7ff6e01c5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:37:41.397 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:41 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:41.098+0000 7ff6e01c5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:37:41.398 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:41 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:41.192+0000 7ff6e01c5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:37:41.398 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:41 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:41.395+0000 7ff6e01c5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:37:41.733 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:41 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:41.586+0000 7ff6e01c5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:37:42.155 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:41 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:41.882+0000 7ff6e01c5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:37:42.155 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:42 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:42.152+0000 7ff6e01c5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:37:42.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:42 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:42.396+0000 7ff6e01c5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:37:43.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:43 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:43.230+0000 7ff6e01c5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:37:44.153 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:43 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:43.824+0000 7ff6e01c5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:37:44.153 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:43 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:43.927+0000 7ff6e01c5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:37:44.153 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:37:44.150+0000 7ff6e01c5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:37:44.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: Active manager daemon y restarted 2024-02-14T17:37:44.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: Activating manager daemon y 2024-02-14T17:37:44.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: osdmap e2: 0 total, 0 up, 0 in 2024-02-14T17:37:44.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: mgrmap e6: y(active, starting, since 0.00391407s) 2024-02-14T17:37:44.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:37:44.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:37:44.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:37:44.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:37:44.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:37:44.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: Manager daemon y is now available 2024-02-14T17:37:44.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:37:44.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:37:44.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:37:44.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:44.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:44 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:45 smithi027 ceph-mon[25828]: Found migration_current of "None". Setting to last migration. 2024-02-14T17:37:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:45 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:45 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:37:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:45 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:37:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:45 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:37:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:45 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:37:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:45 smithi027 ceph-mon[25828]: mgrmap e7: y(active, since 1.0051s) 2024-02-14T17:37:45.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-02-14T17:37:45.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-14T17:37:45.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-14T17:37:45.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-02-14T17:37:45.715 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 5 is available 2024-02-14T17:37:45.715 INFO:teuthology.orchestra.run.smithi027.stdout:Setting orchestrator backend to cephadm... 2024-02-14T17:37:46.454 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:46 smithi027 ceph-mon[25828]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-14T17:37:46.455 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:46 smithi027 ceph-mon[25828]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-14T17:37:47.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:47 smithi027 ceph-mon[25828]: mgrmap e8: y(active, since 2s) 2024-02-14T17:37:47.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:47 smithi027 ceph-mon[25828]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:37:47.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:47 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:37:47.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:47 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:47.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:47 smithi027 ceph-mon[25828]: [14/Feb/2024:17:37:46] ENGINE Bus STARTING 2024-02-14T17:37:47.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:47 smithi027 ceph-mon[25828]: [14/Feb/2024:17:37:46] ENGINE Serving on https://172.21.15.27:7150 2024-02-14T17:37:47.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:47 smithi027 ceph-mon[25828]: [14/Feb/2024:17:37:46] ENGINE Serving on http://172.21.15.27:8765 2024-02-14T17:37:47.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:47 smithi027 ceph-mon[25828]: [14/Feb/2024:17:37:46] ENGINE Bus STARTED 2024-02-14T17:37:47.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:47 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:48.277 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-14T17:37:48.278 INFO:teuthology.orchestra.run.smithi027.stdout:Generating ssh key... 2024-02-14T17:37:48.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:48 smithi027 ceph-mon[25828]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:37:49.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-mon[25828]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:37:49.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-mon[25828]: Generating ssh key... 2024-02-14T17:37:49.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: Generating public/private rsa key pair. 2024-02-14T17:37:49.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: Your identification has been saved in /tmp/tmpk7l8c65x/key. 2024-02-14T17:37:49.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: Your public key has been saved in /tmp/tmpk7l8c65x/key.pub. 2024-02-14T17:37:49.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: The key fingerprint is: 2024-02-14T17:37:49.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: SHA256:IaVzdv/eFmtOUaLrLnQKdA7I36DZ8yI/T1ookRiHkhU ceph-932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:37:49.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: The key's randomart image is: 2024-02-14T17:37:49.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: +---[RSA 3072]----+ 2024-02-14T17:37:49.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: | E. . | 2024-02-14T17:37:49.485 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: | o . o | 2024-02-14T17:37:49.485 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: | o o.=.+ . . .| 2024-02-14T17:37:49.485 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: | . +o*+o.. . ..| 2024-02-14T17:37:49.485 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: | . o*S* o . | 2024-02-14T17:37:49.485 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: | o.=.+ .o ..| 2024-02-14T17:37:49.485 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: | . .=oo. ..o| 2024-02-14T17:37:49.486 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: | ..o++. ..+.| 2024-02-14T17:37:49.486 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: | oo+.oo +o.| 2024-02-14T17:37:49.486 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:37:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: +----[SHA256]-----+ 2024-02-14T17:37:50.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:50 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:37:50.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:50 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:37:51.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZMeCU1ngRvSVVDfL1QqPH7afE4lB61cpJzGTUjqg+ihGNcAsyPx5mFg9uFoxT9upGALOQCJkyheGYTsxNoMke9ngpUeNZxKl/Qr5BTEXF3quTBeYphP2D2AXcFt6lhd25UpkXvB2Lh8BkPnWpINMoSOXcgYaEfqI8xMJ0Cg0ncHWlYnf8uNKH2OVscvCsLph8mJs/lb9yBlZC3rXbyGc3wueQft93vcTQmur1fTj3lEc55+FTXvs9LnUdXNGs7zeAgBY/4olHDt5qyh5fMIHHqscULwSymCo8tjcHuLN5czb/9Yd0HodsAGnvhvqJ3OuEfBVUax/AuWTjGwGNiWLflgkyN1jq3t3kkp1w3NCFxA8t8oRm20zb+o+cK5vPds9LDzYxH++EbLE5WNbr0ecz68l2M8dR6XcQ505metTh7AhK/OgTDGvMOSUKsDXvnSTiW7dT7Q5NPIZeeIBGRIo0Ima2NJdViXb2kIUw5vUqBESCG3YCZ28K7hoEpx5fqsU= ceph-932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:37:51.189 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-14T17:37:51.189 INFO:teuthology.orchestra.run.smithi027.stdout:Adding key to root@localhost authorized_keys... 2024-02-14T17:37:51.189 INFO:teuthology.orchestra.run.smithi027.stdout:Adding host smithi027... 2024-02-14T17:37:51.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:51 smithi027 ceph-mon[25828]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:37:52.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:52 smithi027 ceph-mon[25828]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:37:53.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:53 smithi027 ceph-mon[25828]: Deploying cephadm binary to smithi027 2024-02-14T17:37:56.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Added host 'smithi027' with addr '172.21.15.27' 2024-02-14T17:37:56.938 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mon service... 2024-02-14T17:37:57.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:57 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:37:57.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:57 smithi027 ceph-mon[25828]: Added host smithi027 2024-02-14T17:37:57.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:57 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:58.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-14T17:37:58.440 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mgr service... 2024-02-14T17:37:58.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:58 smithi027 ceph-mon[25828]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:37:58.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:58 smithi027 ceph-mon[25828]: Saving service mon spec with placement count:5 2024-02-14T17:37:58.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:37:58 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:38:00.582 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:00 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:38:00.582 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:00 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:38:00.582 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:00 smithi027 ceph-mon[25828]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:38:00.582 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:00 smithi027 ceph-mon[25828]: Saving service mgr spec with placement count:2 2024-02-14T17:38:00.582 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:00 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:38:00.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-14T17:38:02.601 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:02 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/3671193448' entity='client.admin' 2024-02-14T17:38:03.498 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling the dashboard module... 2024-02-14T17:38:03.754 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:03 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:38:03.754 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:03 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:38:03.754 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:03 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:38:03.755 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:03 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' 2024-02-14T17:38:03.755 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:03 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi027", "caps": []}]: dispatch 2024-02-14T17:38:03.755 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:03 smithi027 ceph-mon[25828]: from='mgr.14120 172.21.15.27:0/660265883' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi027", "caps": []}]': finished 2024-02-14T17:38:03.755 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:03 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/718871683' entity='client.admin' 2024-02-14T17:38:04.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:04 smithi027 ceph-mon[25828]: Deploying daemon agent.smithi027 on smithi027 2024-02-14T17:38:04.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:04 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/1220170344' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-14T17:38:05.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:05 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: ignoring --setuser ceph since I am not root 2024-02-14T17:38:05.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:05 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: ignoring --setgroup ceph since I am not root 2024-02-14T17:38:05.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:05 smithi027 ceph-mon[25828]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:05.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:05 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/1220170344' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-14T17:38:05.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:05 smithi027 ceph-mon[25828]: mgrmap e9: y(active, since 21s) 2024-02-14T17:38:06.024 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:05 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:05.770+0000 7fd3d326b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:38:06.024 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:05 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:05.871+0000 7fd3d326b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:38:06.807 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:06 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/723756621' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-14T17:38:06.807 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:06 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:06.555+0000 7fd3d326b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:38:06.808 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:06 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:06.653+0000 7fd3d326b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:38:06.808 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:06 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:06.805+0000 7fd3d326b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:38:07.114 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:06 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:06.906+0000 7fd3d326b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:38:07.114 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:06 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:06.993+0000 7fd3d326b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:38:07.118 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-02-14T17:38:07.118 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-14T17:38:07.119 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-14T17:38:07.119 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-14T17:38:07.119 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-14T17:38:07.119 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-02-14T17:38:07.119 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-02-14T17:38:07.119 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 9... 2024-02-14T17:38:07.393 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:07 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:07.276+0000 7fd3d326b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:38:07.393 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:07 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:07.391+0000 7fd3d326b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:38:07.733 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:07 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:07.487+0000 7fd3d326b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:38:07.733 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:07 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:07.587+0000 7fd3d326b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:38:08.233 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:07 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:07.772+0000 7fd3d326b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:38:09.233 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:08 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:08.785+0000 7fd3d326b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:38:09.233 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:08 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:08.921+0000 7fd3d326b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:38:09.563 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:09 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:09.265+0000 7fd3d326b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:38:09.563 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:09 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:09.359+0000 7fd3d326b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:38:09.563 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:09 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:09.562+0000 7fd3d326b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:38:09.983 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:09 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:09.755+0000 7fd3d326b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:38:10.320 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:10 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:10.050+0000 7fd3d326b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:38:10.320 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:10 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:10.319+0000 7fd3d326b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:38:10.733 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:10 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:10.564+0000 7fd3d326b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:38:11.733 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:11 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:11.406+0000 7fd3d326b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:38:12.341 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:12.012+0000 7fd3d326b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:38:12.341 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:12.114+0000 7fd3d326b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:38:12.341 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:38:12.339+0000 7fd3d326b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:38:12.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-mon[25828]: Active manager daemon y restarted 2024-02-14T17:38:12.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-mon[25828]: Activating manager daemon y 2024-02-14T17:38:12.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-mon[25828]: osdmap e3: 0 total, 0 up, 0 in 2024-02-14T17:38:12.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-mon[25828]: mgrmap e10: y(active, starting, since 0.00328199s) 2024-02-14T17:38:12.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:38:12.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:38:12.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:38:12.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:38:12.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:38:12.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-mon[25828]: Manager daemon y is now available 2024-02-14T17:38:12.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:12.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:38:12.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:38:13.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:13 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:38:13.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:13 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:38:13.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:13 smithi027 ceph-mon[25828]: [14/Feb/2024:17:38:12] ENGINE Bus STARTING 2024-02-14T17:38:13.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:13 smithi027 ceph-mon[25828]: [14/Feb/2024:17:38:12] ENGINE Serving on https://172.21.15.27:7150 2024-02-14T17:38:13.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:13 smithi027 ceph-mon[25828]: [14/Feb/2024:17:38:12] ENGINE Serving on http://172.21.15.27:8765 2024-02-14T17:38:13.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:13 smithi027 ceph-mon[25828]: [14/Feb/2024:17:38:12] ENGINE Bus STARTED 2024-02-14T17:38:13.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:13 smithi027 ceph-mon[25828]: mgrmap e11: y(active, since 1.00589s) 2024-02-14T17:38:13.968 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-02-14T17:38:13.968 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-14T17:38:13.968 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-14T17:38:13.969 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-02-14T17:38:13.969 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 9 is available 2024-02-14T17:38:13.969 INFO:teuthology.orchestra.run.smithi027.stdout:Generating a dashboard self-signed certificate... 2024-02-14T17:38:14.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:14 smithi027 ceph-mon[25828]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-14T17:38:14.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:14 smithi027 ceph-mon[25828]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-14T17:38:15.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-14T17:38:15.409 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial admin user... 2024-02-14T17:38:15.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:15 smithi027 ceph-mon[25828]: mgrmap e12: y(active, since 2s) 2024-02-14T17:38:15.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:15 smithi027 ceph-mon[25828]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:38:15.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:15 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:15.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:15 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:38:15.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:15 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:15.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:15 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi027", "caps": []}]: dispatch 2024-02-14T17:38:15.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:15 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:15.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:15 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:16.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:16 smithi027 ceph-mon[25828]: Deploying daemon agent.smithi027 on smithi027 2024-02-14T17:38:16.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:16 smithi027 ceph-mon[25828]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:38:17.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$x20oj5JJobvdOchavspvmefWnB4hgC8mk2XO6miQsp0ThsAKZEr82", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707932296, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-14T17:38:17.032 INFO:teuthology.orchestra.run.smithi027.stdout:Fetching dashboard port number... 2024-02-14T17:38:17.710 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:18.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 8443 2024-02-14T17:38:18.290 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-14T17:38:18.290 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-02-14T17:38:18.298 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-14T17:38:18.298 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-02-14T17:38:18.298 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-02-14T17:38:18.298 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-14T17:38:18.300 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph Dashboard is now available at: 2024-02-14T17:38:18.301 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:38:18.301 INFO:teuthology.orchestra.run.smithi027.stdout: URL: https://smithi027.front.sepia.ceph.com:8443/ 2024-02-14T17:38:18.301 INFO:teuthology.orchestra.run.smithi027.stdout: User: admin 2024-02-14T17:38:18.301 INFO:teuthology.orchestra.run.smithi027.stdout: Password: u8nfec4gqn 2024-02-14T17:38:18.301 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:38:18.301 INFO:teuthology.orchestra.run.smithi027.stdout:Saving cluster configuration to /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config directory 2024-02-14T17:38:18.302 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling autotune for osd_memory_target 2024-02-14T17:38:18.562 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:18 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/2834169517' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-14T17:38:19.600 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:19 smithi027 ceph-mon[25828]: mgrmap e13: y(active, since 6s) 2024-02-14T17:38:20.974 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-14T17:38:20.974 INFO:teuthology.orchestra.run.smithi027.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-14T17:38:20.974 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:38:20.974 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout:Or, if you are only running a single cluster on this host: 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout: ceph telemetry on 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout:For more information see: 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:38:20.975 INFO:teuthology.orchestra.run.smithi027.stdout:Bootstrap complete. 2024-02-14T17:38:21.023 INFO:tasks.cephadm:Fetching config... 2024-02-14T17:38:21.023 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:38:21.023 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-14T17:38:21.039 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-14T17:38:21.039 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:38:21.039 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-14T17:38:21.094 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-14T17:38:21.095 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:38:21.095 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-14T17:38:21.246 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-14T17:38:21.246 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:38:21.246 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-14T17:38:21.304 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-14T17:38:21.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZMeCU1ngRvSVVDfL1QqPH7afE4lB61cpJzGTUjqg+ihGNcAsyPx5mFg9uFoxT9upGALOQCJkyheGYTsxNoMke9ngpUeNZxKl/Qr5BTEXF3quTBeYphP2D2AXcFt6lhd25UpkXvB2Lh8BkPnWpINMoSOXcgYaEfqI8xMJ0Cg0ncHWlYnf8uNKH2OVscvCsLph8mJs/lb9yBlZC3rXbyGc3wueQft93vcTQmur1fTj3lEc55+FTXvs9LnUdXNGs7zeAgBY/4olHDt5qyh5fMIHHqscULwSymCo8tjcHuLN5czb/9Yd0HodsAGnvhvqJ3OuEfBVUax/AuWTjGwGNiWLflgkyN1jq3t3kkp1w3NCFxA8t8oRm20zb+o+cK5vPds9LDzYxH++EbLE5WNbr0ecz68l2M8dR6XcQ505metTh7AhK/OgTDGvMOSUKsDXvnSTiW7dT7Q5NPIZeeIBGRIo0Ima2NJdViXb2kIUw5vUqBESCG3YCZ28K7hoEpx5fqsU= ceph-932ae096-cb5f-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-14T17:38:21.468 INFO:teuthology.orchestra.run.smithi027.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZMeCU1ngRvSVVDfL1QqPH7afE4lB61cpJzGTUjqg+ihGNcAsyPx5mFg9uFoxT9upGALOQCJkyheGYTsxNoMke9ngpUeNZxKl/Qr5BTEXF3quTBeYphP2D2AXcFt6lhd25UpkXvB2Lh8BkPnWpINMoSOXcgYaEfqI8xMJ0Cg0ncHWlYnf8uNKH2OVscvCsLph8mJs/lb9yBlZC3rXbyGc3wueQft93vcTQmur1fTj3lEc55+FTXvs9LnUdXNGs7zeAgBY/4olHDt5qyh5fMIHHqscULwSymCo8tjcHuLN5czb/9Yd0HodsAGnvhvqJ3OuEfBVUax/AuWTjGwGNiWLflgkyN1jq3t3kkp1w3NCFxA8t8oRm20zb+o+cK5vPds9LDzYxH++EbLE5WNbr0ecz68l2M8dR6XcQ505metTh7AhK/OgTDGvMOSUKsDXvnSTiW7dT7Q5NPIZeeIBGRIo0Ima2NJdViXb2kIUw5vUqBESCG3YCZ28K7hoEpx5fqsU= ceph-932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:38:21.486 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZMeCU1ngRvSVVDfL1QqPH7afE4lB61cpJzGTUjqg+ihGNcAsyPx5mFg9uFoxT9upGALOQCJkyheGYTsxNoMke9ngpUeNZxKl/Qr5BTEXF3quTBeYphP2D2AXcFt6lhd25UpkXvB2Lh8BkPnWpINMoSOXcgYaEfqI8xMJ0Cg0ncHWlYnf8uNKH2OVscvCsLph8mJs/lb9yBlZC3rXbyGc3wueQft93vcTQmur1fTj3lEc55+FTXvs9LnUdXNGs7zeAgBY/4olHDt5qyh5fMIHHqscULwSymCo8tjcHuLN5czb/9Yd0HodsAGnvhvqJ3OuEfBVUax/AuWTjGwGNiWLflgkyN1jq3t3kkp1w3NCFxA8t8oRm20zb+o+cK5vPds9LDzYxH++EbLE5WNbr0ecz68l2M8dR6XcQ505metTh7AhK/OgTDGvMOSUKsDXvnSTiW7dT7Q5NPIZeeIBGRIo0Ima2NJdViXb2kIUw5vUqBESCG3YCZ28K7hoEpx5fqsU= ceph-932ae096-cb5f-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-14T17:38:21.513 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:21 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/3642847987' entity='client.admin' 2024-02-14T17:38:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZMeCU1ngRvSVVDfL1QqPH7afE4lB61cpJzGTUjqg+ihGNcAsyPx5mFg9uFoxT9upGALOQCJkyheGYTsxNoMke9ngpUeNZxKl/Qr5BTEXF3quTBeYphP2D2AXcFt6lhd25UpkXvB2Lh8BkPnWpINMoSOXcgYaEfqI8xMJ0Cg0ncHWlYnf8uNKH2OVscvCsLph8mJs/lb9yBlZC3rXbyGc3wueQft93vcTQmur1fTj3lEc55+FTXvs9LnUdXNGs7zeAgBY/4olHDt5qyh5fMIHHqscULwSymCo8tjcHuLN5czb/9Yd0HodsAGnvhvqJ3OuEfBVUax/AuWTjGwGNiWLflgkyN1jq3t3kkp1w3NCFxA8t8oRm20zb+o+cK5vPds9LDzYxH++EbLE5WNbr0ecz68l2M8dR6XcQ505metTh7AhK/OgTDGvMOSUKsDXvnSTiW7dT7Q5NPIZeeIBGRIo0Ima2NJdViXb2kIUw5vUqBESCG3YCZ28K7hoEpx5fqsU= ceph-932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:38:21.547 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZMeCU1ngRvSVVDfL1QqPH7afE4lB61cpJzGTUjqg+ihGNcAsyPx5mFg9uFoxT9upGALOQCJkyheGYTsxNoMke9ngpUeNZxKl/Qr5BTEXF3quTBeYphP2D2AXcFt6lhd25UpkXvB2Lh8BkPnWpINMoSOXcgYaEfqI8xMJ0Cg0ncHWlYnf8uNKH2OVscvCsLph8mJs/lb9yBlZC3rXbyGc3wueQft93vcTQmur1fTj3lEc55+FTXvs9LnUdXNGs7zeAgBY/4olHDt5qyh5fMIHHqscULwSymCo8tjcHuLN5czb/9Yd0HodsAGnvhvqJ3OuEfBVUax/AuWTjGwGNiWLflgkyN1jq3t3kkp1w3NCFxA8t8oRm20zb+o+cK5vPds9LDzYxH++EbLE5WNbr0ecz68l2M8dR6XcQ505metTh7AhK/OgTDGvMOSUKsDXvnSTiW7dT7Q5NPIZeeIBGRIo0Ima2NJdViXb2kIUw5vUqBESCG3YCZ28K7hoEpx5fqsU= ceph-932ae096-cb5f-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-14T17:38:21.594 INFO:teuthology.orchestra.run.smithi096.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZMeCU1ngRvSVVDfL1QqPH7afE4lB61cpJzGTUjqg+ihGNcAsyPx5mFg9uFoxT9upGALOQCJkyheGYTsxNoMke9ngpUeNZxKl/Qr5BTEXF3quTBeYphP2D2AXcFt6lhd25UpkXvB2Lh8BkPnWpINMoSOXcgYaEfqI8xMJ0Cg0ncHWlYnf8uNKH2OVscvCsLph8mJs/lb9yBlZC3rXbyGc3wueQft93vcTQmur1fTj3lEc55+FTXvs9LnUdXNGs7zeAgBY/4olHDt5qyh5fMIHHqscULwSymCo8tjcHuLN5czb/9Yd0HodsAGnvhvqJ3OuEfBVUax/AuWTjGwGNiWLflgkyN1jq3t3kkp1w3NCFxA8t8oRm20zb+o+cK5vPds9LDzYxH++EbLE5WNbr0ecz68l2M8dR6XcQ505metTh7AhK/OgTDGvMOSUKsDXvnSTiW7dT7Q5NPIZeeIBGRIo0Ima2NJdViXb2kIUw5vUqBESCG3YCZ28K7hoEpx5fqsU= ceph-932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:38:21.609 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-14T17:38:21.998 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:38:22.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:22.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:22.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:22.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:38:22.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:22.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:22.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:22.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:22.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:22.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:38:22.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:22.736 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:24.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:24.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:24 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/4180592056' entity='client.admin' 2024-02-14T17:38:24.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:38:24.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:24.563 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-14T17:38:24.564 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-14T17:38:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:38:29.374 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:29 smithi027 ceph-mon[25828]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:38:29.375 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:29 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:29.375 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:29 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:38:29.375 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:29 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:29.375 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:29 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:38:29.375 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:29 smithi027 ceph-mon[25828]: Updating smithi027:/etc/ceph/ceph.conf 2024-02-14T17:38:29.375 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:29 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:29.375 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:29 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:29.375 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:29 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:29.375 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:29 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:29.696 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi040 2024-02-14T17:38:29.696 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-02-14T17:38:29.696 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/etc/ceph/ceph.conf 2024-02-14T17:38:29.714 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-02-14T17:38:29.714 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:38:29.771 INFO:tasks.cephadm:Adding host smithi040 to orchestrator... 2024-02-14T17:38:29.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch host add smithi040 2024-02-14T17:38:30.019 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:38:30.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:30 smithi027 ceph-mon[25828]: Updating smithi027:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:38:30.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:30 smithi027 ceph-mon[25828]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:38:30.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:30 smithi027 ceph-mon[25828]: Updating smithi027:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:38:30.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:30 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:30.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:30 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:32.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:32 smithi027 ceph-mon[25828]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi040", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:38:33.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:33 smithi027 ceph-mon[25828]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:33.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:33 smithi027 ceph-mon[25828]: Deploying cephadm binary to smithi040 2024-02-14T17:38:35.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:35 smithi027 ceph-mon[25828]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:36.374 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi040' with addr '172.21.15.40' 2024-02-14T17:38:37.151 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch host ls --format=json 2024-02-14T17:38:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:38:37.680 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:37 smithi027 ceph-mon[25828]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:37.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:37 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:37.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:37 smithi027 ceph-mon[25828]: Added host smithi040 2024-02-14T17:38:37.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:37 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:38:37.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:37 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:38.601 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:38 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:38.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:38 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:38.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:38 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:38:38.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:38 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:38.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:38 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:38:38.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:38 smithi027 ceph-mon[25828]: Updating smithi040:/etc/ceph/ceph.conf 2024-02-14T17:38:38.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:38 smithi027 ceph-mon[25828]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:38:38.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:38 smithi027 ceph-mon[25828]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:38:38.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:38 smithi027 ceph-mon[25828]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:38:38.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:38 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:38.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:38 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:38.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:38 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi040", "caps": []}]: dispatch 2024-02-14T17:38:38.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:38 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi040", "caps": []}]': finished 2024-02-14T17:38:39.358 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:38:39.358 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}] 2024-02-14T17:38:39.665 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:39 smithi027 ceph-mon[25828]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:39.961 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi096 2024-02-14T17:38:39.962 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-02-14T17:38:39.962 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/etc/ceph/ceph.conf 2024-02-14T17:38:39.980 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-02-14T17:38:39.980 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:38:40.038 INFO:tasks.cephadm:Adding host smithi096 to orchestrator... 2024-02-14T17:38:40.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch host add smithi096 2024-02-14T17:38:40.288 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:38:41.380 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: Deploying daemon agent.smithi040 on smithi040 2024-02-14T17:38:41.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:38:41.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:41.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:41.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:41.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:38:41.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:41.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:38:41.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:41.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:38:41.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:41.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:38:41.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:41.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:41.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:41.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:41 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:42.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:42 smithi027 ceph-mon[25828]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:43.474 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:43 smithi027 ceph-mon[25828]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi096", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:38:43.475 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:43 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:44.330 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:44 smithi027 ceph-mon[25828]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:44.330 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:44 smithi027 ceph-mon[25828]: Deploying cephadm binary to smithi096 2024-02-14T17:38:44.330 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:44 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:44.330 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:44 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:44.330 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:44 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:44.330 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:44 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:44.331 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:44 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:46.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:46 smithi027 ceph-mon[25828]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:46.639 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi096' with addr '172.21.15.96' 2024-02-14T17:38:47.303 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch host ls --format=json 2024-02-14T17:38:47.554 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:38:47.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:47 smithi027 ceph-mon[25828]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:47.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:47 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:47.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:47 smithi027 ceph-mon[25828]: Added host smithi096 2024-02-14T17:38:47.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:47 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:38:47.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:47 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:48.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:48 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:48.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:48 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:48.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:48 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:38:48.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:48 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:48.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:48 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:38:48.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:48 smithi027 ceph-mon[25828]: Updating smithi096:/etc/ceph/ceph.conf 2024-02-14T17:38:48.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:48 smithi027 ceph-mon[25828]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:38:48.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:48 smithi027 ceph-mon[25828]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:38:48.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:48 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:48.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:48 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:48.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:48 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi096", "caps": []}]: dispatch 2024-02-14T17:38:48.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:48 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi096", "caps": []}]': finished 2024-02-14T17:38:49.359 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:38:49.359 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.96", "hostname": "smithi096", "labels": [], "status": ""}] 2024-02-14T17:38:49.971 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:49 smithi027 ceph-mon[25828]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:38:49.971 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:49 smithi027 ceph-mon[25828]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:49.971 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:49 smithi027 ceph-mon[25828]: Deploying daemon agent.smithi096 on smithi096 2024-02-14T17:38:49.971 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:49 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:49.998 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-14T17:38:49.998 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph osd crush tunables default 2024-02-14T17:38:50.250 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:38:50.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:38:50.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:50.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:50.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:38:50.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:50.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:38:50.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:50.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:38:50.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:50.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:38:50.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:50.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:50.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:50.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:52.198 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-02-14T17:38:52.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:52 smithi027 ceph-mon[25828]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:52.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:52 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/1445905676' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-14T17:38:52.864 INFO:tasks.cephadm:Adding mon.a on smithi027 2024-02-14T17:38:52.864 INFO:tasks.cephadm:Adding mon.b on smithi040 2024-02-14T17:38:52.865 INFO:tasks.cephadm:Adding mon.c on smithi096 2024-02-14T17:38:52.865 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch apply mon '3;smithi027:172.21.15.27=a;smithi040:172.21.15.40=b;smithi096:172.21.15.96=c' 2024-02-14T17:38:53.155 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:38:53.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:53 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/1445905676' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-14T17:38:53.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:53 smithi027 ceph-mon[25828]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:38:53.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:53 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:54.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:54 smithi027 ceph-mon[25828]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:54.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:54 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:54.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:54 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:54.971 INFO:teuthology.orchestra.run.smithi096.stdout:Scheduled mon update... 2024-02-14T17:38:55.557 DEBUG:teuthology.orchestra.run.smithi040:mon.b> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.b.service 2024-02-14T17:38:55.560 DEBUG:teuthology.orchestra.run.smithi096:mon.c> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.c.service 2024-02-14T17:38:55.562 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-14T17:38:55.562 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-14T17:38:56.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:55 smithi027 ceph-mon[25828]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:55 smithi027 ceph-mon[25828]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi040:172.21.15.40=b;smithi096:172.21.15.96=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:38:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:55 smithi027 ceph-mon[25828]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi040:172.21.15.40=b;smithi096:172.21.15.96=c;count:3 2024-02-14T17:38:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:38:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:38:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:38:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:38:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:55 smithi027 ceph-mon[25828]: Deploying daemon mon.c on smithi096 2024-02-14T17:38:56.338 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:38:58.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:57 smithi027 ceph-mon[25828]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:38:58.532 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:38:58.533 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":1,"fsid":"932ae096-cb5f-11ee-95b9-87774f69a715","modified":"2024-02-14T17:37:02.237562Z","created":"2024-02-14T17:37:02.237562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-14T17:38:58.535 INFO:teuthology.orchestra.run.smithi096.stderr:dumped monmap epoch 1 2024-02-14T17:38:59.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:58 smithi027 ceph-mon[25828]: from='client.? 172.21.15.96:0/338962890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-14T17:39:00.165 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-14T17:39:00.165 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-14T17:39:00.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:38:59 smithi027 ceph-mon[25828]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:00.404 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.c/config 2024-02-14T17:39:01.195 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 podman[27716]: 2024-02-14 17:39:01.194378583 +0000 UTC m=+0.437497872 container create ef7f0c7cea0a8ccaa91232dd383c00878012c2820be5e78e552b8eb3069fa65c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-14T17:39:02.143 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 podman[27716]: 2024-02-14 17:39:01.940804118 +0000 UTC m=+1.183923412 container init ef7f0c7cea0a8ccaa91232dd383c00878012c2820be5e78e552b8eb3069fa65c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-02-14T17:39:02.143 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 podman[27716]: 2024-02-14 17:39:01.946791214 +0000 UTC m=+1.189910503 container start ef7f0c7cea0a8ccaa91232dd383c00878012c2820be5e78e552b8eb3069fa65c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-02-14T17:39:02.143 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: set uid:gid to 167:167 (ceph:ceph) 2024-02-14T17:39:02.143 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev), process ceph-mon, pid 7 2024-02-14T17:39:02.143 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pidfile_write: ignore empty --pid-file 2024-02-14T17:39:02.143 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: load: jerasure load: lrc 2024-02-14T17:39:02.143 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: RocksDB version: 7.9.2 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Git sha 0 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Compile date 2024-02-14 11:59:27 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: DB SUMMARY 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: DB Session ID: S6HRD3SV9R2PAOQ20JYJ 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: CURRENT file: CURRENT 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: IDENTITY file: IDENTITY 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.error_if_exists: 0 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.create_if_missing: 0 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.paranoid_checks: 1 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-14T17:39:02.144 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-14T17:39:02.145 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-14T17:39:02.145 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.env: 0x55fdade2eae0 2024-02-14T17:39:02.145 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.fs: PosixFileSystem 2024-02-14T17:39:02.145 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.info_log: 0x55fdaec4f8c0 2024-02-14T17:39:02.145 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_file_opening_threads: 16 2024-02-14T17:39:02.146 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.statistics: (nil) 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.use_fsync: 0 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_log_file_size: 0 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.keep_log_file_num: 1000 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.recycle_log_file_num: 0 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.allow_fallocate: 1 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.allow_mmap_reads: 0 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.allow_mmap_writes: 0 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.use_direct_reads: 0 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.create_missing_column_families: 0 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.db_log_dir: 2024-02-14T17:39:02.147 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.wal_dir: 2024-02-14T17:39:02.148 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-14T17:39:02.148 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-14T17:39:02.148 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-14T17:39:02.148 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-14T17:39:02.148 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-14T17:39:02.148 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-14T17:39:02.148 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.advise_random_on_open: 1 2024-02-14T17:39:02.148 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.db_write_buffer_size: 0 2024-02-14T17:39:02.148 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.write_buffer_manager: 0x55fdaec23400 2024-02-14T17:39:02.148 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-14T17:39:02.148 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-14T17:39:02.148 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-14T17:39:02.148 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.rate_limiter: (nil) 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.wal_recovery_mode: 2 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.enable_thread_tracking: 0 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.enable_pipelined_write: 0 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.unordered_write: 0 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.row_cache: None 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.wal_filter: None 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.allow_ingest_behind: 0 2024-02-14T17:39:02.149 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.two_write_queues: 0 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.manual_wal_flush: 0 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.wal_compression: 0 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.atomic_flush: 0 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.log_readahead_size: 0 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.best_efforts_recovery: 0 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.allow_data_in_errors: 0 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.db_host_id: __hostname__ 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-14T17:39:02.150 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_background_jobs: 2 2024-02-14T17:39:02.151 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_background_compactions: -1 2024-02-14T17:39:02.151 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_subcompactions: 1 2024-02-14T17:39:02.151 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-14T17:39:02.151 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-14T17:39:02.151 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-14T17:39:02.151 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_total_wal_size: 0 2024-02-14T17:39:02.151 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-14T17:39:02.151 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-14T17:39:02.151 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-14T17:39:02.151 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-14T17:39:02.151 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_open_files: -1 2024-02-14T17:39:02.151 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.bytes_per_sync: 0 2024-02-14T17:39:02.152 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-14T17:39:02.152 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-14T17:39:02.152 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compaction_readahead_size: 0 2024-02-14T17:39:02.152 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_background_flushes: -1 2024-02-14T17:39:02.152 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Compression algorithms supported: 2024-02-14T17:39:02.152 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-14T17:39:02.152 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: kZSTD supported: 0 2024-02-14T17:39:02.152 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: kXpressCompression supported: 0 2024-02-14T17:39:02.152 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: kLZ4HCCompression supported: 1 2024-02-14T17:39:02.152 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: kZlibCompression supported: 1 2024-02-14T17:39:02.152 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: kSnappyCompression supported: 1 2024-02-14T17:39:02.152 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: kLZ4Compression supported: 1 2024-02-14T17:39:02.152 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: kBZip2Compression supported: 0 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.merge_operator: 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compaction_filter: None 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compaction_filter_factory: None 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.sst_partitioner_factory: None 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fdaec4f9c0) 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout: cache_index_and_filter_blocks: 1 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-14T17:39:02.153 INFO:journalctl@ceph.mon.c.smithi096.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: pin_top_level_index_and_filter: 1 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: index_type: 0 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: data_block_index_type: 0 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: index_shortening: 1 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: checksum: 4 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: no_block_cache: 0 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: block_cache: 0x55fdaec53090 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: block_cache_name: BinnedLRUCache 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: block_cache_options: 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: capacity : 536870912 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: num_shard_bits : 4 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: strict_capacity_limit : 0 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: high_pri_pool_ratio: 0.000 2024-02-14T17:39:02.154 INFO:journalctl@ceph.mon.c.smithi096.stdout: block_cache_compressed: (nil) 2024-02-14T17:39:02.155 INFO:journalctl@ceph.mon.c.smithi096.stdout: persistent_cache: (nil) 2024-02-14T17:39:02.155 INFO:journalctl@ceph.mon.c.smithi096.stdout: block_size: 4096 2024-02-14T17:39:02.155 INFO:journalctl@ceph.mon.c.smithi096.stdout: block_size_deviation: 10 2024-02-14T17:39:02.155 INFO:journalctl@ceph.mon.c.smithi096.stdout: block_restart_interval: 16 2024-02-14T17:39:02.155 INFO:journalctl@ceph.mon.c.smithi096.stdout: index_block_restart_interval: 1 2024-02-14T17:39:02.155 INFO:journalctl@ceph.mon.c.smithi096.stdout: metadata_block_size: 4096 2024-02-14T17:39:02.155 INFO:journalctl@ceph.mon.c.smithi096.stdout: partition_filters: 0 2024-02-14T17:39:02.155 INFO:journalctl@ceph.mon.c.smithi096.stdout: use_delta_encoding: 1 2024-02-14T17:39:02.156 INFO:journalctl@ceph.mon.c.smithi096.stdout: filter_policy: bloomfilter 2024-02-14T17:39:02.156 INFO:journalctl@ceph.mon.c.smithi096.stdout: whole_key_filtering: 1 2024-02-14T17:39:02.156 INFO:journalctl@ceph.mon.c.smithi096.stdout: verify_compression: 0 2024-02-14T17:39:02.156 INFO:journalctl@ceph.mon.c.smithi096.stdout: read_amp_bytes_per_bit: 0 2024-02-14T17:39:02.156 INFO:journalctl@ceph.mon.c.smithi096.stdout: format_version: 5 2024-02-14T17:39:02.156 INFO:journalctl@ceph.mon.c.smithi096.stdout: enable_index_compression: 1 2024-02-14T17:39:02.156 INFO:journalctl@ceph.mon.c.smithi096.stdout: block_align: 0 2024-02-14T17:39:02.156 INFO:journalctl@ceph.mon.c.smithi096.stdout: max_auto_readahead_size: 262144 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout: prepopulate_block_cache: 0 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout: initial_auto_readahead_size: 8192 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout: num_file_reads_for_auto_readahead: 2 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.write_buffer_size: 33554432 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_write_buffer_number: 2 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compression: NoCompression 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.bottommost_compression: Disabled 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.prefix_extractor: nullptr 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.num_levels: 7 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-14T17:39:02.157 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-14T17:39:02.158 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-14T17:39:02.158 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-14T17:39:02.158 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:39:02.158 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-14T17:39:02.158 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-14T17:39:02.158 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-14T17:39:02.158 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-14T17:39:02.158 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-14T17:39:02.158 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compression_opts.level: 32767 2024-02-14T17:39:02.158 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compression_opts.strategy: 0 2024-02-14T17:39:02.158 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compression_opts.enabled: false 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.target_file_size_base: 67108864 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-14T17:39:02.159 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.arena_block_size: 1048576 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.disable_auto_compactions: 0 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-14T17:39:02.160 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.table_properties_collectors: 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.inplace_update_support: 0 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.bloom_locality: 0 2024-02-14T17:39:02.161 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.max_successive_merges: 0 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.paranoid_file_checks: 0 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.force_consistency_checks: 1 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.report_bg_io_stats: 0 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.ttl: 2592000 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.enable_blob_files: false 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.min_blob_size: 0 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.blob_file_size: 268435456 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-14T17:39:02.162 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.blob_file_starting_level: 0 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8ab19886-7dba-45ea-8a63-8a7461bbc365 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932341977122, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932341977536, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707932341, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8ab19886-7dba-45ea-8a63-8a7461bbc365", "db_session_id": "S6HRD3SV9R2PAOQ20JYJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932341977616, "job": 1, "event": "recovery_finished"} 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fdaed44000 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: DB pointer 0x55fdaed2e000 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-14T17:39:02.163 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: ** DB Stats ** 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: ** Compaction Stats [default] ** 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-14T17:39:02.164 INFO:journalctl@ceph.mon.c.smithi096.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-14T17:39:02.165 INFO:journalctl@ceph.mon.c.smithi096.stdout: 2024-02-14T17:39:02.165 INFO:journalctl@ceph.mon.c.smithi096.stdout: ** Compaction Stats [default] ** 2024-02-14T17:39:02.165 INFO:journalctl@ceph.mon.c.smithi096.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-14T17:39:02.165 INFO:journalctl@ceph.mon.c.smithi096.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:39:02.165 INFO:journalctl@ceph.mon.c.smithi096.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-14T17:39:02.165 INFO:journalctl@ceph.mon.c.smithi096.stdout: 2024-02-14T17:39:02.165 INFO:journalctl@ceph.mon.c.smithi096.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-14T17:39:02.165 INFO:journalctl@ceph.mon.c.smithi096.stdout: 2024-02-14T17:39:02.165 INFO:journalctl@ceph.mon.c.smithi096.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:39:02.165 INFO:journalctl@ceph.mon.c.smithi096.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-14T17:39:02.165 INFO:journalctl@ceph.mon.c.smithi096.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout: Block cache BinnedLRUCache@0x55fdaec53090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout: 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: using public_addr v2:172.21.15.96:0/0 -> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: starting mon.c rank -1 at public addrs [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] at bind addrs [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(???) e0 preinit fsid 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(synchronizing).mds e1 new map 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout: e1 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout: legacy client fscid: -1 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout: 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout: No filesystems configured 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-14T17:39:02.167 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Added host smithi040 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Updating smithi040:/etc/ceph/ceph.conf 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:39:02.168 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi040", "caps": []}]: dispatch 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi040", "caps": []}]': finished 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Deploying daemon agent.smithi040 on smithi040 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:02.169 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi096", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Deploying cephadm binary to smithi096 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.170 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Added host smithi096 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Updating smithi096:/etc/ceph/ceph.conf 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:39:02.171 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:39:02.172 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.172 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.172 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi096", "caps": []}]: dispatch 2024-02-14T17:39:02.172 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi096", "caps": []}]': finished 2024-02-14T17:39:02.172 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:39:02.172 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:02.172 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Deploying daemon agent.smithi096 on smithi096 2024-02-14T17:39:02.172 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.172 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:39:02.172 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.172 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:02.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:02.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:02.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:02.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:02.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/1445905676' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/1445905676' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi040:172.21.15.40=b;smithi096:172.21.15.96=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:39:02.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi040:172.21.15.40=b;smithi096:172.21.15.96=c;count:3 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: Deploying daemon mon.c on smithi096 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: from='client.? 172.21.15.96:0/338962890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:01 smithi096 ceph-mon[27796]: mon.c@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-02-14T17:39:02.175 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:02 smithi096 bash[27716]: ef7f0c7cea0a8ccaa91232dd383c00878012c2820be5e78e552b8eb3069fa65c 2024-02-14T17:39:02.176 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:02 smithi096 systemd[1]: Started Ceph mon.c for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:39:02.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:01 smithi027 ceph-mon[25828]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:07.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:39:07.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: mon.a calling monitor election 2024-02-14T17:39:07.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:07.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:07.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:07.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:07.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: mon.c calling monitor election 2024-02-14T17:39:07.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: monmap e2: 2 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],c=[v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: fsmap 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: mgrmap e13: y(active, since 54s) 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: overall HEALTH_OK 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:07.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:07 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:07.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:39:07.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: mon.a calling monitor election 2024-02-14T17:39:07.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:07.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:07.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:07.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:07.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: mon.c calling monitor election 2024-02-14T17:39:07.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:07.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:07.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:07.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:07.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-14T17:39:07.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: monmap e2: 2 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],c=[v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: fsmap 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: mgrmap e13: y(active, since 54s) 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: overall HEALTH_OK 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:07.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:07.548 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:07 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:08.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:08 smithi027 ceph-mon[25828]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:08.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:08 smithi027 ceph-mon[25828]: Deploying daemon mon.b on smithi040 2024-02-14T17:39:08.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:08 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:08.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:08 smithi096 ceph-mon[27796]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:08.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:08 smithi096 ceph-mon[27796]: Deploying daemon mon.b on smithi040 2024-02-14T17:39:08.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:08 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:08.762 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:39:08.762 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":2,"fsid":"932ae096-cb5f-11ee-95b9-87774f69a715","modified":"2024-02-14T17:39:02.000601Z","created":"2024-02-14T17:37:02.237562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-14T17:39:08.764 INFO:teuthology.orchestra.run.smithi096.stderr:dumped monmap epoch 2 2024-02-14T17:39:09.347 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:09 smithi096 ceph-mon[27796]: from='client.? 172.21.15.96:0/1574323952' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-14T17:39:09.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:09 smithi027 ceph-mon[25828]: from='client.? 172.21.15.96:0/1574323952' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-14T17:39:10.374 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-14T17:39:10.374 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-14T17:39:10.402 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:10 smithi096 ceph-mon[27796]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:10.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:10 smithi027 ceph-mon[25828]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.036 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: load: jerasure load: lrc 2024-02-14T17:39:11.158 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.c/config 2024-02-14T17:39:11.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: RocksDB version: 7.9.2 2024-02-14T17:39:11.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Git sha 0 2024-02-14T17:39:11.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Compile date 2024-02-14 11:59:27 2024-02-14T17:39:11.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: DB SUMMARY 2024-02-14T17:39:11.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: DB Session ID: 5TCJ0Q991LPSFEUGREB6 2024-02-14T17:39:11.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: CURRENT file: CURRENT 2024-02-14T17:39:11.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: IDENTITY file: IDENTITY 2024-02-14T17:39:11.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-14T17:39:11.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-14T17:39:11.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-02-14T17:39:11.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.error_if_exists: 0 2024-02-14T17:39:11.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.create_if_missing: 0 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.paranoid_checks: 1 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.env: 0x561067fc4ae0 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.fs: PosixFileSystem 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.info_log: 0x5610690438c0 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_file_opening_threads: 16 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.statistics: (nil) 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.use_fsync: 0 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_log_file_size: 0 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-14T17:39:11.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.keep_log_file_num: 1000 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.recycle_log_file_num: 0 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.allow_fallocate: 1 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.allow_mmap_reads: 0 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.allow_mmap_writes: 0 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.use_direct_reads: 0 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.create_missing_column_families: 0 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.db_log_dir: 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.wal_dir: 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-14T17:39:11.482 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.advise_random_on_open: 1 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.db_write_buffer_size: 0 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.write_buffer_manager: 0x561069017400 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.rate_limiter: (nil) 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.wal_recovery_mode: 2 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.enable_thread_tracking: 0 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.enable_pipelined_write: 0 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.unordered_write: 0 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-14T17:39:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-14T17:39:11.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-14T17:39:11.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-14T17:39:11.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.row_cache: None 2024-02-14T17:39:11.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.wal_filter: None 2024-02-14T17:39:11.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-14T17:39:11.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.allow_ingest_behind: 0 2024-02-14T17:39:11.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.two_write_queues: 0 2024-02-14T17:39:11.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.manual_wal_flush: 0 2024-02-14T17:39:11.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.wal_compression: 0 2024-02-14T17:39:11.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.atomic_flush: 0 2024-02-14T17:39:11.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-14T17:39:11.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-14T17:39:11.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-14T17:39:11.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.log_readahead_size: 0 2024-02-14T17:39:11.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-14T17:39:11.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.best_efforts_recovery: 0 2024-02-14T17:39:11.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-14T17:39:11.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-14T17:39:11.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.allow_data_in_errors: 0 2024-02-14T17:39:11.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.db_host_id: __hostname__ 2024-02-14T17:39:11.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-14T17:39:11.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_background_jobs: 2 2024-02-14T17:39:11.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_background_compactions: -1 2024-02-14T17:39:11.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_subcompactions: 1 2024-02-14T17:39:11.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-14T17:39:11.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-14T17:39:11.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-14T17:39:11.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_total_wal_size: 0 2024-02-14T17:39:11.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-14T17:39:11.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-14T17:39:11.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-14T17:39:11.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-14T17:39:11.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_open_files: -1 2024-02-14T17:39:11.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.bytes_per_sync: 0 2024-02-14T17:39:11.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-14T17:39:11.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-14T17:39:11.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compaction_readahead_size: 0 2024-02-14T17:39:11.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_background_flushes: -1 2024-02-14T17:39:11.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Compression algorithms supported: 2024-02-14T17:39:11.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-14T17:39:11.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: kZSTD supported: 0 2024-02-14T17:39:11.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: kXpressCompression supported: 0 2024-02-14T17:39:11.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: kLZ4HCCompression supported: 1 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: kZlibCompression supported: 1 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: kSnappyCompression supported: 1 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: kLZ4Compression supported: 1 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: kBZip2Compression supported: 0 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.merge_operator: 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compaction_filter: None 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compaction_filter_factory: None 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.sst_partitioner_factory: None 2024-02-14T17:39:11.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-14T17:39:11.490 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-14T17:39:11.490 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5610690439c0) 2024-02-14T17:39:11.490 INFO:journalctl@ceph.mon.b.smithi040.stdout: cache_index_and_filter_blocks: 1 2024-02-14T17:39:11.490 INFO:journalctl@ceph.mon.b.smithi040.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-14T17:39:11.490 INFO:journalctl@ceph.mon.b.smithi040.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-14T17:39:11.490 INFO:journalctl@ceph.mon.b.smithi040.stdout: pin_top_level_index_and_filter: 1 2024-02-14T17:39:11.490 INFO:journalctl@ceph.mon.b.smithi040.stdout: index_type: 0 2024-02-14T17:39:11.490 INFO:journalctl@ceph.mon.b.smithi040.stdout: data_block_index_type: 0 2024-02-14T17:39:11.490 INFO:journalctl@ceph.mon.b.smithi040.stdout: index_shortening: 1 2024-02-14T17:39:11.490 INFO:journalctl@ceph.mon.b.smithi040.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-14T17:39:11.490 INFO:journalctl@ceph.mon.b.smithi040.stdout: checksum: 4 2024-02-14T17:39:11.490 INFO:journalctl@ceph.mon.b.smithi040.stdout: no_block_cache: 0 2024-02-14T17:39:11.490 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_cache: 0x561069047090 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_cache_name: BinnedLRUCache 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_cache_options: 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: capacity : 536870912 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: num_shard_bits : 4 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: strict_capacity_limit : 0 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: high_pri_pool_ratio: 0.000 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_cache_compressed: (nil) 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: persistent_cache: (nil) 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_size: 4096 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_size_deviation: 10 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_restart_interval: 16 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: index_block_restart_interval: 1 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: metadata_block_size: 4096 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: partition_filters: 0 2024-02-14T17:39:11.491 INFO:journalctl@ceph.mon.b.smithi040.stdout: use_delta_encoding: 1 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout: filter_policy: bloomfilter 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout: whole_key_filtering: 1 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout: verify_compression: 0 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout: read_amp_bytes_per_bit: 0 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout: format_version: 5 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout: enable_index_compression: 1 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_align: 0 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout: max_auto_readahead_size: 262144 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout: prepopulate_block_cache: 0 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout: initial_auto_readahead_size: 8192 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout: num_file_reads_for_auto_readahead: 2 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.write_buffer_size: 33554432 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_write_buffer_number: 2 2024-02-14T17:39:11.492 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compression: NoCompression 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.bottommost_compression: Disabled 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.prefix_extractor: nullptr 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.num_levels: 7 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-14T17:39:11.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compression_opts.level: 32767 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compression_opts.strategy: 0 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compression_opts.enabled: false 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.target_file_size_base: 67108864 2024-02-14T17:39:11.494 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-14T17:39:11.495 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.arena_block_size: 1048576 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.disable_auto_compactions: 0 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.table_properties_collectors: 2024-02-14T17:39:11.496 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.inplace_update_support: 0 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.bloom_locality: 0 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.max_successive_merges: 0 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.paranoid_file_checks: 0 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.force_consistency_checks: 1 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.report_bg_io_stats: 0 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.ttl: 2592000 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-14T17:39:11.497 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.enable_blob_files: false 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.min_blob_size: 0 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.blob_file_size: 268435456 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.blob_file_starting_level: 0 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fc305ed6-a5c1-4ec2-bf5e-490dce6e0a83 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932351036896, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-14T17:39:11.498 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932351037315, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707932351, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fc305ed6-a5c1-4ec2-bf5e-490dce6e0a83", "db_session_id": "5TCJ0Q991LPSFEUGREB6", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932351037396, "job": 1, "event": "recovery_finished"} 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561069138000 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: DB pointer 0x561069122000 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout: ** DB Stats ** 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:39:11.499 INFO:journalctl@ceph.mon.b.smithi040.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: ** Compaction Stats [default] ** 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: ** Compaction Stats [default] ** 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: 2024-02-14T17:39:11.500 INFO:journalctl@ceph.mon.b.smithi040.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: Block cache BinnedLRUCache@0x561069047090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-14T17:39:11.501 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: using public_addr v2:172.21.15.40:0/0 -> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: starting mon.b rank -1 at public addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] at bind addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(???) e0 preinit fsid 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(synchronizing).mds e1 new map 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(synchronizing).mds e1 print_map 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout: e1 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout: legacy client fscid: -1 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout: 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout: No filesystems configured 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-14T17:39:11.502 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Added host smithi040 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:39:11.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Updating smithi040:/etc/ceph/ceph.conf 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi040", "caps": []}]: dispatch 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi040", "caps": []}]': finished 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Deploying daemon agent.smithi040 on smithi040 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.504 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi096", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:39:11.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Deploying cephadm binary to smithi096 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Added host smithi096 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.507 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:39:11.507 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:11.507 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:11.507 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Updating smithi096:/etc/ceph/ceph.conf 2024-02-14T17:39:11.507 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:39:11.507 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:39:11.507 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.507 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.507 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi096", "caps": []}]: dispatch 2024-02-14T17:39:11.507 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi096", "caps": []}]': finished 2024-02-14T17:39:11.507 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:39:11.507 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.507 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Deploying daemon agent.smithi096 on smithi096 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.508 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/1445905676' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/1445905676' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi040:172.21.15.40=b;smithi096:172.21.15.96=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi040:172.21.15.40=b;smithi096:172.21.15.96=c;count:3 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:11.509 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Deploying daemon mon.c on smithi096 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='client.? 172.21.15.96:0/338962890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.a calling monitor election 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:11.510 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.c calling monitor election 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: monmap e2: 2 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],c=[v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: fsmap 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mgrmap e13: y(active, since 54s) 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: overall HEALTH_OK 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.511 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.512 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.512 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.512 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:11.512 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:11.512 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:11.512 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.512 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: Deploying daemon mon.b on smithi040 2024-02-14T17:39:11.512 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:11.512 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: from='client.? 172.21.15.96:0/1574323952' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-14T17:39:11.512 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:11.512 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 ceph-mon[27571]: mon.b@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-02-14T17:39:11.512 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 bash[27533]: 544f97a89dd67c7cef5be4a9578afdf4cd9820d59c42fc93304d3f901d7ecc03 2024-02-14T17:39:11.512 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:11 smithi040 systemd[1]: Started Ceph mon.b for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:39:16.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:16.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:39:16.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: mon.c calling monitor election 2024-02-14T17:39:16.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: mon.a calling monitor election 2024-02-14T17:39:16.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:16.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:16.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:16.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:16.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:16.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:16.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:16.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:16.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:16.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-14T17:39:16.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:39:16.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: fsmap 2024-02-14T17:39:16.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:39:16.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: mgrmap e13: y(active, since 63s) 2024-02-14T17:39:16.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-14T17:39:16.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-14T17:39:16.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-14T17:39:16.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: mon.b (rank 2) addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] is down (out of quorum) 2024-02-14T17:39:16.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:16 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.508 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:16.508 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:39:16.508 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: mon.c calling monitor election 2024-02-14T17:39:16.508 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: mon.a calling monitor election 2024-02-14T17:39:16.508 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:16.508 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:16.508 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:16.508 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:16.508 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: fsmap 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: mgrmap e13: y(active, since 63s) 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: mon.b (rank 2) addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] is down (out of quorum) 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.509 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.510 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.510 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.510 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.510 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.510 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:16 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:16.624 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-02-14T17:39:16.624 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":3,"fsid":"932ae096-cb5f-11ee-95b9-87774f69a715","modified":"2024-02-14T17:39:11.060477Z","created":"2024-02-14T17:37:02.237562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-14T17:39:16.625 INFO:teuthology.orchestra.run.smithi096.stderr:dumped monmap epoch 3 2024-02-14T17:39:17.240 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-14T17:39:17.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph config generate-minimal-conf 2024-02-14T17:39:17.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:17.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: Updating smithi027:/etc/ceph/ceph.conf 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: Updating smithi040:/etc/ceph/ceph.conf 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: Updating smithi096:/etc/ceph/ceph.conf 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: Updating smithi027:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='client.? 172.21.15.96:0/3963841680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.548 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.548 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.548 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:39:17.548 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: Reconfiguring mon.a (unknown last config time)... 2024-02-14T17:39:17.548 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:17.548 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:39:17.548 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:17.548 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: Reconfiguring daemon mon.a on smithi027 2024-02-14T17:39:17.548 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:17.575 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.575 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.575 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.575 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:17.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:17.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:17.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: Updating smithi027:/etc/ceph/ceph.conf 2024-02-14T17:39:17.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: Updating smithi040:/etc/ceph/ceph.conf 2024-02-14T17:39:17.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: Updating smithi096:/etc/ceph/ceph.conf 2024-02-14T17:39:17.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:17.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: Updating smithi027:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:39:17.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:39:17.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:39:17.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='client.? 172.21.15.96:0/3963841680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-14T17:39:17.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:17.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:39:17.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: Reconfiguring mon.a (unknown last config time)... 2024-02-14T17:39:17.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:17.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:39:17.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:17.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: Reconfiguring daemon mon.a on smithi027 2024-02-14T17:39:17.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:18.143 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: mon.c calling monitor election 2024-02-14T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: mon.a calling monitor election 2024-02-14T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: fsmap 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: mgrmap e13: y(active, since 63s) 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: mon.b (rank 2) addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] is down (out of quorum) 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: Updating smithi027:/etc/ceph/ceph.conf 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: Updating smithi040:/etc/ceph/ceph.conf 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: Updating smithi096:/etc/ceph/ceph.conf 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: Updating smithi027:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='client.? 172.21.15.96:0/3963841680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: Reconfiguring mon.a (unknown last config time)... 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:39:18.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:18.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: Reconfiguring daemon mon.a on smithi027 2024-02-14T17:39:18.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:19.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:18 smithi096 ceph-mon[27796]: mon.b calling monitor election 2024-02-14T17:39:19.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:18 smithi096 ceph-mon[27796]: mon.b calling monitor election 2024-02-14T17:39:19.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:18 smithi096 ceph-mon[27796]: mon.c calling monitor election 2024-02-14T17:39:19.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:18 smithi096 ceph-mon[27796]: mon.a calling monitor election 2024-02-14T17:39:19.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:18 smithi096 ceph-mon[27796]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:39:19.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:18 smithi096 ceph-mon[27796]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:39:19.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:18 smithi096 ceph-mon[27796]: fsmap 2024-02-14T17:39:19.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:18 smithi096 ceph-mon[27796]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:39:19.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:18 smithi096 ceph-mon[27796]: mgrmap e13: y(active, since 65s) 2024-02-14T17:39:19.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:18 smithi096 ceph-mon[27796]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-14T17:39:19.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:18 smithi096 ceph-mon[27796]: Cluster is now healthy 2024-02-14T17:39:19.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:18 smithi096 ceph-mon[27796]: overall HEALTH_OK 2024-02-14T17:39:19.061 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:18 smithi027 ceph-mon[25828]: mon.b calling monitor election 2024-02-14T17:39:19.061 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:18 smithi027 ceph-mon[25828]: mon.b calling monitor election 2024-02-14T17:39:19.061 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:18 smithi027 ceph-mon[25828]: mon.c calling monitor election 2024-02-14T17:39:19.061 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:18 smithi027 ceph-mon[25828]: mon.a calling monitor election 2024-02-14T17:39:19.061 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:18 smithi027 ceph-mon[25828]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:39:19.062 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:18 smithi027 ceph-mon[25828]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:39:19.062 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:18 smithi027 ceph-mon[25828]: fsmap 2024-02-14T17:39:19.062 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:18 smithi027 ceph-mon[25828]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:39:19.062 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:18 smithi027 ceph-mon[25828]: mgrmap e13: y(active, since 65s) 2024-02-14T17:39:19.062 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:18 smithi027 ceph-mon[25828]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-14T17:39:19.062 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:18 smithi027 ceph-mon[25828]: Cluster is now healthy 2024-02-14T17:39:19.062 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:18 smithi027 ceph-mon[25828]: overall HEALTH_OK 2024-02-14T17:39:19.178 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: mon.b calling monitor election 2024-02-14T17:39:19.178 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: mon.b calling monitor election 2024-02-14T17:39:19.178 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: mon.c calling monitor election 2024-02-14T17:39:19.178 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: mon.a calling monitor election 2024-02-14T17:39:19.178 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:39:19.178 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:39:19.178 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: fsmap 2024-02-14T17:39:19.178 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:39:19.179 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: mgrmap e13: y(active, since 65s) 2024-02-14T17:39:19.179 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-14T17:39:19.179 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: Cluster is now healthy 2024-02-14T17:39:19.179 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:18 smithi040 ceph-mon[27571]: overall HEALTH_OK 2024-02-14T17:39:19.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:19 smithi027 ceph-mon[25828]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:19.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:19 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:19.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:19 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:19.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:19 smithi027 ceph-mon[25828]: Reconfiguring mon.b (monmap changed)... 2024-02-14T17:39:19.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:19 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:19.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:19 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:39:19.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:19 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:19.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:19 smithi027 ceph-mon[25828]: Reconfiguring daemon mon.b on smithi040 2024-02-14T17:39:19.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:19 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:19.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:19 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:19.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:19 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:19.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:19 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:20.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:19 smithi096 ceph-mon[27796]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:20.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:19 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:20.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:19 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:20.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:19 smithi096 ceph-mon[27796]: Reconfiguring mon.b (monmap changed)... 2024-02-14T17:39:20.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:19 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:20.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:19 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:39:20.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:19 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:20.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:19 smithi096 ceph-mon[27796]: Reconfiguring daemon mon.b on smithi040 2024-02-14T17:39:20.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:19 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:20.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:19 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:20.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:19 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:20.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:19 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:20.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:19 smithi040 ceph-mon[27571]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:20.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:19 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:20.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:19 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:20.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:19 smithi040 ceph-mon[27571]: Reconfiguring mon.b (monmap changed)... 2024-02-14T17:39:20.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:19 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:20.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:19 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:39:20.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:19 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:20.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:19 smithi040 ceph-mon[27571]: Reconfiguring daemon mon.b on smithi040 2024-02-14T17:39:20.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:19 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:39:20.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:19 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:20.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:19 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:20.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:19 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:20.233 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:39:20 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:39:20.061+0000 7fd389466700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-14T17:39:20.421 INFO:teuthology.orchestra.run.smithi027.stdout:# minimal ceph.conf for 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:39:20.421 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-02-14T17:39:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:39:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: mon_host = [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] 2024-02-14T17:39:21.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:20 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:21.046 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:20 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:21.046 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:20 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:21.046 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:20 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:21.046 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:20 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/2088996766' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:21.078 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-14T17:39:21.078 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:39:21.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.conf 2024-02-14T17:39:21.114 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:39:21.114 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:39:21.187 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-02-14T17:39:21.187 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.conf 2024-02-14T17:39:21.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:20 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:21.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:20 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:21.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:20 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:21.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:20 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:21.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:20 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/2088996766' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:21.219 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-02-14T17:39:21.219 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:39:21.287 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-02-14T17:39:21.288 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.conf 2024-02-14T17:39:21.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:20 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:21.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:20 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:21.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:20 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:21.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:20 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:21.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:20 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/2088996766' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:21.324 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-02-14T17:39:21.324 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:39:21.395 INFO:tasks.cephadm:Adding mgr.y on smithi027 2024-02-14T17:39:21.395 INFO:tasks.cephadm:Adding mgr.x on smithi040 2024-02-14T17:39:21.395 INFO:tasks.cephadm:Adding mgr.z on smithi096 2024-02-14T17:39:21.395 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch apply mgr '3;smithi027=y;smithi040=x;smithi096=z' 2024-02-14T17:39:21.681 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.c/config 2024-02-14T17:39:22.106 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:21 smithi027 ceph-mon[25828]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:22.106 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:21 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:22.106 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:21 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:22.106 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:21 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:22.106 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:21 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:39:22.106 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:21 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:22.110 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:21 smithi096 ceph-mon[27796]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:22.110 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:21 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:22.111 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:21 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:22.111 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:21 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:22.111 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:21 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:39:22.111 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:21 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:22.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:21 smithi040 ceph-mon[27571]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:22.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:21 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:22.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:21 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:22.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:21 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:39:22.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:21 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:39:22.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:21 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:22.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:22 smithi027 ceph-mon[25828]: Reconfiguring mon.c (monmap changed)... 2024-02-14T17:39:22.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:22 smithi027 ceph-mon[25828]: Reconfiguring daemon mon.c on smithi096 2024-02-14T17:39:23.198 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:22 smithi096 ceph-mon[27796]: Reconfiguring mon.c (monmap changed)... 2024-02-14T17:39:23.199 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:22 smithi096 ceph-mon[27796]: Reconfiguring daemon mon.c on smithi096 2024-02-14T17:39:23.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:22 smithi040 ceph-mon[27571]: Reconfiguring mon.c (monmap changed)... 2024-02-14T17:39:23.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:22 smithi040 ceph-mon[27571]: Reconfiguring daemon mon.c on smithi096 2024-02-14T17:39:23.965 INFO:teuthology.orchestra.run.smithi096.stdout:Scheduled mgr update... 2024-02-14T17:39:24.081 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:23 smithi027 ceph-mon[25828]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:24.206 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:23 smithi096 ceph-mon[27796]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:24.228 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:23 smithi040 ceph-mon[27571]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:24.798 DEBUG:teuthology.orchestra.run.smithi040:mgr.x> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mgr.x.service 2024-02-14T17:39:24.800 DEBUG:teuthology.orchestra.run.smithi096:mgr.z> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mgr.z.service 2024-02-14T17:39:24.802 INFO:tasks.cephadm:Deploying OSDs... 2024-02-14T17:39:24.802 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:39:24.802 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-02-14T17:39:24.822 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-14T17:39:24.822 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/nvme1n1 2024-02-14T17:39:24.881 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/nvme1n1 2024-02-14T17:39:24.881 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-14T17:39:24.881 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 763 Links: 1 Device type: 103,2 2024-02-14T17:39:24.882 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-14T17:39:24.882 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-14T17:39:24.882 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-02-14 17:39:23.677302587 +0000 2024-02-14T17:39:24.882 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-02-14 17:36:20.708106299 +0000 2024-02-14T17:39:24.882 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-02-14 17:36:20.708106299 +0000 2024-02-14T17:39:24.882 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-02-14 17:36:20.698106559 +0000 2024-02-14T17:39:24.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-02-14T17:39:24.957 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-02-14T17:39:24.957 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-02-14T17:39:24.957 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000317557 s, 1.6 MB/s 2024-02-14T17:39:24.958 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-02-14T17:39:25.017 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/nvme2n1 2024-02-14T17:39:25.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:24 smithi096 ceph-mon[27796]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=y;smithi040=x;smithi096=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:39:25.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:24 smithi096 ceph-mon[27796]: Saving service mgr spec with placement smithi027=y;smithi040=x;smithi096=z;count:3 2024-02-14T17:39:25.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:24 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:25.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:24 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:25.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:24 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:25.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:24 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:25.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:24 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:25.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:24 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:25.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:24 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:25.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:24 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:39:25.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:24 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-14T17:39:25.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:24 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:39:25.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:24 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:25.075 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/nvme2n1 2024-02-14T17:39:25.075 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-14T17:39:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-02-14T17:39:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-14T17:39:25.075 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-14T17:39:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-02-14 17:39:23.688302302 +0000 2024-02-14T17:39:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-02-14 17:36:20.842102820 +0000 2024-02-14T17:39:25.076 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-02-14 17:36:20.842102820 +0000 2024-02-14T17:39:25.076 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-02-14 17:36:20.830103132 +0000 2024-02-14T17:39:25.076 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-02-14T17:39:25.130 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:24 smithi040 ceph-mon[27571]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=y;smithi040=x;smithi096=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:39:25.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:24 smithi040 ceph-mon[27571]: Saving service mgr spec with placement smithi027=y;smithi040=x;smithi096=z;count:3 2024-02-14T17:39:25.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:24 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:25.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:24 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:25.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:24 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:25.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:24 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:25.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:24 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:25.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:24 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:25.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:24 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:25.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:24 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:39:25.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:24 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-14T17:39:25.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:24 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:39:25.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:24 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:25.146 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-02-14T17:39:25.146 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-02-14T17:39:25.146 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.0002441 s, 2.1 MB/s 2024-02-14T17:39:25.148 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-02-14T17:39:25.206 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/nvme3n1 2024-02-14T17:39:25.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:24 smithi027 ceph-mon[25828]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=y;smithi040=x;smithi096=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:39:25.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:24 smithi027 ceph-mon[25828]: Saving service mgr spec with placement smithi027=y;smithi040=x;smithi096=z;count:3 2024-02-14T17:39:25.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:25.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:25.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:25.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:25.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:25.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:25.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:25.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:39:25.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-14T17:39:25.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:39:25.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:25.249 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/nvme3n1 2024-02-14T17:39:25.249 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-14T17:39:25.250 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-02-14T17:39:25.250 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-14T17:39:25.250 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-14T17:39:25.250 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-02-14 17:39:23.699302018 +0000 2024-02-14T17:39:25.250 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-02-14 17:36:20.975099368 +0000 2024-02-14T17:39:25.250 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-02-14 17:36:20.975099368 +0000 2024-02-14T17:39:25.250 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-02-14 17:36:20.963099679 +0000 2024-02-14T17:39:25.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-02-14T17:39:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-02-14T17:39:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-02-14T17:39:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000227055 s, 2.3 MB/s 2024-02-14T17:39:25.322 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-02-14T17:39:25.379 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/nvme4n1 2024-02-14T17:39:25.434 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/nvme4n1 2024-02-14T17:39:25.435 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-14T17:39:25.435 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-02-14T17:39:25.435 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-14T17:39:25.435 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-14T17:39:25.435 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-02-14 17:39:23.710301733 +0000 2024-02-14T17:39:25.435 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-02-14 17:36:21.109095889 +0000 2024-02-14T17:39:25.435 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-02-14 17:36:21.109095889 +0000 2024-02-14T17:39:25.435 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-02-14 17:36:21.097096201 +0000 2024-02-14T17:39:25.435 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-02-14T17:39:25.504 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-02-14T17:39:25.504 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-02-14T17:39:25.504 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000227712 s, 2.2 MB/s 2024-02-14T17:39:25.505 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-02-14T17:39:25.562 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-02-14T17:39:25.563 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-02-14T17:39:25.579 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-14T17:39:25.579 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/nvme1n1 2024-02-14T17:39:25.634 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/nvme1n1 2024-02-14T17:39:25.634 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-14T17:39:25.634 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 763 Links: 1 Device type: 103,2 2024-02-14T17:39:25.635 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-14T17:39:25.635 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-14T17:39:25.635 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-02-14 17:39:19.117040001 +0000 2024-02-14T17:39:25.635 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-02-14 17:36:22.277498524 +0000 2024-02-14T17:39:25.635 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-02-14 17:36:22.277498524 +0000 2024-02-14T17:39:25.635 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-02-14 17:36:22.269497831 +0000 2024-02-14T17:39:25.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-02-14T17:39:25.701 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-02-14T17:39:25.701 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-02-14T17:39:25.701 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000284914 s, 1.8 MB/s 2024-02-14T17:39:25.702 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-02-14T17:39:25.759 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/nvme2n1 2024-02-14T17:39:25.813 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/nvme2n1 2024-02-14T17:39:25.813 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-14T17:39:25.813 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-02-14T17:39:25.813 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-14T17:39:25.813 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-14T17:39:25.813 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-02-14 17:39:19.128039850 +0000 2024-02-14T17:39:25.814 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-02-14 17:36:22.420498084 +0000 2024-02-14T17:39:25.814 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-02-14 17:36:22.420498084 +0000 2024-02-14T17:39:25.814 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-02-14 17:36:22.415498154 +0000 2024-02-14T17:39:25.814 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-02-14T17:39:25.879 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-02-14T17:39:25.879 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-02-14T17:39:25.879 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000200641 s, 2.6 MB/s 2024-02-14T17:39:25.880 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-02-14T17:39:25.935 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/nvme3n1 2024-02-14T17:39:25.990 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/nvme3n1 2024-02-14T17:39:25.990 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-14T17:39:25.990 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-02-14T17:39:25.990 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-14T17:39:25.991 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-14T17:39:25.991 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-02-14 17:39:19.139039697 +0000 2024-02-14T17:39:25.991 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-02-14 17:36:22.568496008 +0000 2024-02-14T17:39:25.991 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-02-14 17:36:22.568496008 +0000 2024-02-14T17:39:25.991 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-02-14 17:36:22.560496120 +0000 2024-02-14T17:39:25.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-02-14T17:39:26.055 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-02-14T17:39:26.055 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-02-14T17:39:26.056 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000202659 s, 2.5 MB/s 2024-02-14T17:39:26.057 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-02-14T17:39:26.112 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/nvme4n1 2024-02-14T17:39:26.166 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/nvme4n1 2024-02-14T17:39:26.167 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-14T17:39:26.167 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 800 Links: 1 Device type: 103,8 2024-02-14T17:39:26.167 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-14T17:39:26.167 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-14T17:39:26.167 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-02-14 17:39:19.150039545 +0000 2024-02-14T17:39:26.167 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-02-14 17:36:22.717493917 +0000 2024-02-14T17:39:26.167 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-02-14 17:36:22.717493917 +0000 2024-02-14T17:39:26.167 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-02-14 17:36:22.709494029 +0000 2024-02-14T17:39:26.167 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-02-14T17:39:26.177 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:25 smithi096 ceph-mon[27796]: Deploying daemon mgr.z on smithi096 2024-02-14T17:39:26.177 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:25 smithi096 ceph-mon[27796]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:26.228 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:25 smithi040 ceph-mon[27571]: Deploying daemon mgr.z on smithi096 2024-02-14T17:39:26.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:25 smithi040 ceph-mon[27571]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-02-14T17:39:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-02-14T17:39:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000203535 s, 2.5 MB/s 2024-02-14T17:39:26.233 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-02-14T17:39:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:25 smithi027 ceph-mon[25828]: Deploying daemon mgr.z on smithi096 2024-02-14T17:39:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:25 smithi027 ceph-mon[25828]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:26.288 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-02-14T17:39:26.289 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-02-14T17:39:26.304 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-14T17:39:26.305 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/nvme1n1 2024-02-14T17:39:26.359 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/nvme1n1 2024-02-14T17:39:26.360 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-14T17:39:26.360 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-02-14T17:39:26.360 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-14T17:39:26.360 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-14T17:39:26.360 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-02-14 17:39:18.874752916 +0000 2024-02-14T17:39:26.360 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-02-14 17:36:23.891823144 +0000 2024-02-14T17:39:26.360 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-02-14 17:36:23.891823144 +0000 2024-02-14T17:39:26.360 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-02-14 17:36:23.879823423 +0000 2024-02-14T17:39:26.360 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-02-14T17:39:26.430 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-02-14T17:39:26.430 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-02-14T17:39:26.430 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000237184 s, 2.2 MB/s 2024-02-14T17:39:26.431 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-02-14T17:39:26.488 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/nvme2n1 2024-02-14T17:39:26.543 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/nvme2n1 2024-02-14T17:39:26.543 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-14T17:39:26.543 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-02-14T17:39:26.543 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-14T17:39:26.543 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-14T17:39:26.543 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-02-14 17:39:18.885752661 +0000 2024-02-14T17:39:26.543 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-02-14 17:36:24.044819584 +0000 2024-02-14T17:39:26.544 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-02-14 17:36:24.044819584 +0000 2024-02-14T17:39:26.544 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-02-14 17:36:24.035819794 +0000 2024-02-14T17:39:26.544 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-02-14T17:39:26.610 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-02-14T17:39:26.611 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-02-14T17:39:26.611 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.00021185 s, 2.4 MB/s 2024-02-14T17:39:26.612 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-02-14T17:39:26.667 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/nvme3n1 2024-02-14T17:39:26.723 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/nvme3n1 2024-02-14T17:39:26.724 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-14T17:39:26.724 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-02-14T17:39:26.724 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-14T17:39:26.724 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-14T17:39:26.724 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-02-14 17:39:18.896752405 +0000 2024-02-14T17:39:26.724 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-02-14 17:36:24.199815979 +0000 2024-02-14T17:39:26.724 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-02-14 17:36:24.199815979 +0000 2024-02-14T17:39:26.724 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-02-14 17:36:24.186816281 +0000 2024-02-14T17:39:26.724 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-02-14T17:39:26.791 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-02-14T17:39:26.791 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-02-14T17:39:26.791 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000229187 s, 2.2 MB/s 2024-02-14T17:39:26.792 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-02-14T17:39:26.848 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/nvme4n1 2024-02-14T17:39:26.905 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/nvme4n1 2024-02-14T17:39:26.905 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-14T17:39:26.905 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-02-14T17:39:26.905 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-14T17:39:26.905 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-14T17:39:26.905 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-02-14 17:39:18.906752173 +0000 2024-02-14T17:39:26.905 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-02-14 17:36:24.350812466 +0000 2024-02-14T17:39:26.905 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-02-14 17:36:24.350812466 +0000 2024-02-14T17:39:26.905 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-02-14 17:36:24.341812675 +0000 2024-02-14T17:39:26.906 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-02-14T17:39:26.971 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-02-14T17:39:26.971 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-02-14T17:39:26.972 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000215319 s, 2.4 MB/s 2024-02-14T17:39:26.973 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-02-14T17:39:27.028 INFO:tasks.cephadm:Deploying osd.0 on smithi027 with /dev/nvme4n1... 2024-02-14T17:39:27.028 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-02-14T17:39:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:39:27.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:27 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:27 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:27 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:27 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:27 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:27 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:27 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:39:27.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:27 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-14T17:39:27.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:27 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:39:27.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:27 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:27.412 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:27 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.412 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:27 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.412 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:27 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.412 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:27 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.412 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:27 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.412 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:27 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.412 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:27 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:39:27.412 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:27 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-14T17:39:27.412 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:27 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:39:27.412 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:27 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:27.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:27 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:27 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:27 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:27 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:27 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:27 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:27.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:27 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:39:27.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:27 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-14T17:39:27.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:27 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:39:27.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:27 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:27.795 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:27.502+0000 7f6344d6a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:39:28.347 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:28 smithi040 ceph-mon[27571]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:28.347 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:28 smithi040 ceph-mon[27571]: Deploying daemon mgr.x on smithi040 2024-02-14T17:39:28.422 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:28 smithi027 ceph-mon[25828]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:28.422 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:28 smithi027 ceph-mon[25828]: Deploying daemon mgr.x on smithi040 2024-02-14T17:39:28.462 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:28 smithi096 ceph-mon[27796]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:28.462 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:28 smithi096 ceph-mon[27796]: Deploying daemon mgr.x on smithi040 2024-02-14T17:39:28.462 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:28 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:28.132+0000 7f6344d6a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:39:28.462 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:28 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:28.226+0000 7f6344d6a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:39:28.462 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:28 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:28.366+0000 7f6344d6a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:39:28.795 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:28 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:28.459+0000 7f6344d6a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:39:28.795 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:28 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:28.540+0000 7f6344d6a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:39:29.091 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:28 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:28.800+0000 7f6344d6a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:39:29.091 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:28 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:28.910+0000 7f6344d6a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:39:29.091 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:28 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:28.995+0000 7f6344d6a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:39:29.091 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:29 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:29.088+0000 7f6344d6a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:39:29.229 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:29 smithi040 systemd[1]: Starting Ceph mgr.x for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:39:29.327 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/nvme4n1 2024-02-14T17:39:29.327 INFO:teuthology.orchestra.run.smithi027.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-14T17:39:29.327 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-02-14T17:39:29.327 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-02-14T17:39:29.327 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-02-14T17:39:29.327 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123177 s, 851 MB/s 2024-02-14T17:39:29.327 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-02-14T17:39:29.545 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:29 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:29.262+0000 7f6344d6a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:39:29.552 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:29 smithi040 podman[29227]: 2024-02-14 17:39:29.154884101 +0000 UTC m=+0.017237856 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:39:29.552 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:29 smithi040 podman[29227]: 2024-02-14 17:39:29.25683279 +0000 UTC m=+0.119186588 container create 9e3b48ef91d78206fec8838b2a85cf09d68e15d19133f77dafd9c34f64374c2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, ceph=True, CEPH_POINT_RELEASE=) 2024-02-14T17:39:29.552 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:29 smithi040 podman[29227]: 2024-02-14 17:39:29.44241455 +0000 UTC m=+0.304768315 container init 9e3b48ef91d78206fec8838b2a85cf09d68e15d19133f77dafd9c34f64374c2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-14T17:39:29.552 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:29 smithi040 podman[29227]: 2024-02-14 17:39:29.447915669 +0000 UTC m=+0.310269425 container start 9e3b48ef91d78206fec8838b2a85cf09d68e15d19133f77dafd9c34f64374c2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x, org.label-schema.build-date=20240214, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-14T17:39:29.552 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:29 smithi040 ceph-mgr[29265]: -- 172.21.15.40:0/3982276450 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5652d815c1a0 con 0x5652d8132c00 2024-02-14T17:39:29.552 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:29 smithi040 bash[29227]: 9e3b48ef91d78206fec8838b2a85cf09d68e15d19133f77dafd9c34f64374c2e 2024-02-14T17:39:29.949 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi027:/dev/nvme4n1 2024-02-14T17:39:29.979 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:29 smithi040 systemd[1]: Started Ceph mgr.x for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:39:30.219 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:39:30.295 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:30.251+0000 7f6344d6a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:39:30.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-mon[27796]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:30.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:30.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:30.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:30.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:39:30.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:39:30.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:30.472 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:30 smithi027 ceph-mon[25828]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:30.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:30 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:30 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:30 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:30 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:30 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:30.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:30 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:30.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:30 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:30.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:30 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:30 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:39:30.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:30 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:39:30.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:30 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:30.478 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:30.002+0000 7f9ea15a4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:39:30.479 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:30.098+0000 7f9ea15a4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:39:30.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-mon[27571]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:30.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:30.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:30.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:30.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:30.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:39:30.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:39:30.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:30.634 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:30.386+0000 7f6344d6a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:39:30.918 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:30.726+0000 7f6344d6a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:39:30.918 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:30 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:30.815+0000 7f6344d6a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:39:31.106 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:30.765+0000 7f9ea15a4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:39:31.106 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:30 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:30.863+0000 7f9ea15a4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:39:31.106 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:31 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:31.011+0000 7f9ea15a4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:39:31.222 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:31 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:31.040+0000 7f6344d6a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:39:31.424 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:31 smithi040 ceph-mon[27571]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:39:31.425 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:31 smithi040 ceph-mon[27571]: Reconfiguring mgr.y (unknown last config time)... 2024-02-14T17:39:31.425 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:31 smithi040 ceph-mon[27571]: Reconfiguring daemon mgr.y on smithi027 2024-02-14T17:39:31.425 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:31 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:31.110+0000 7f9ea15a4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:39:31.425 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:31 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:31.197+0000 7f9ea15a4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:39:31.474 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:31 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:31.229+0000 7f6344d6a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:39:31.474 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:31 smithi096 ceph-mon[27796]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:39:31.474 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:31 smithi096 ceph-mon[27796]: Reconfiguring mgr.y (unknown last config time)... 2024-02-14T17:39:31.474 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:31 smithi096 ceph-mon[27796]: Reconfiguring daemon mgr.y on smithi027 2024-02-14T17:39:31.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:31 smithi027 ceph-mon[25828]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:39:31.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:31 smithi027 ceph-mon[25828]: Reconfiguring mgr.y (unknown last config time)... 2024-02-14T17:39:31.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:31 smithi027 ceph-mon[25828]: Reconfiguring daemon mgr.y on smithi027 2024-02-14T17:39:31.688 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:31 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:31.486+0000 7f9ea15a4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:39:31.688 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:31 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:31.599+0000 7f9ea15a4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:39:31.688 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:31 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:31.686+0000 7f9ea15a4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:39:31.795 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:31 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:31.555+0000 7f6344d6a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:39:31.964 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:31 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:31.782+0000 7f9ea15a4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:39:32.149 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:31 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:31.880+0000 7f6344d6a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:39:32.149 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:32 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:32.147+0000 7f6344d6a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:39:32.216 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:31 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:31.962+0000 7f9ea15a4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:39:32.475 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:32 smithi096 ceph-mon[27796]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:32.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:32 smithi040 ceph-mon[27571]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:32.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:32 smithi027 ceph-mon[25828]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:33.159 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:32 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:32.985+0000 7f6344d6a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:39:33.177 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:33.042+0000 7f9ea15a4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:39:33.474 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:33.181+0000 7f9ea15a4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:39:33.475 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.475 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.475 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:33.475 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:33.475 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:33.476 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:33.476 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.476 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:39:33.476 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.476 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.476 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.476 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.477 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:39:33.477 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:39:33.477 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:39:33.477 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:33.477 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.477 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.477 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:39:33.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:39:33.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:39:33.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:33.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.547 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:33.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:33.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:33.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:33.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:39:33.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.646 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.647 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:39:33.647 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:39:33.647 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:39:33.647 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:33.647 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.647 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.647 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:33 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:33.729 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:33.525+0000 7f9ea15a4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:39:33.729 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:33.616+0000 7f9ea15a4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:39:33.877 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:33.561+0000 7f6344d6a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:39:33.877 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:33.658+0000 7f6344d6a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:39:33.877 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:39:33 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:39:33.875+0000 7f6344d6a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:39:34.008 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:33 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:33.815+0000 7f9ea15a4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:39:34.008 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:34.006+0000 7f9ea15a4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:39:34.573 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:34.308+0000 7f9ea15a4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:39:34.863 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:34.571+0000 7f9ea15a4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:39:34.864 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:34.810+0000 7f9ea15a4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:39:34.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:34 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:34.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:34 smithi027 ceph-mon[25828]: Standby manager daemon z started 2024-02-14T17:39:34.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:34 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.96:0/3123834103' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-14T17:39:34.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:34 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.96:0/3123834103' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:39:34.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:34 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.96:0/3123834103' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-14T17:39:34.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:34 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.96:0/3123834103' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:39:34.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:34 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:34.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:34 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:34.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:34 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:34.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:34 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:34.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:34 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:34.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:34 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:34.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:34 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:35.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:35.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-mon[27571]: Standby manager daemon z started 2024-02-14T17:39:35.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.96:0/3123834103' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-14T17:39:35.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.96:0/3123834103' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:39:35.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.96:0/3123834103' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-14T17:39:35.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.96:0/3123834103' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:39:35.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:35.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:35.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:35.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:35.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:35.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:35.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:34 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:35.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:34 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:35.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:34 smithi096 ceph-mon[27796]: Standby manager daemon z started 2024-02-14T17:39:35.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:34 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.96:0/3123834103' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-14T17:39:35.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:34 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.96:0/3123834103' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:39:35.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:34 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.96:0/3123834103' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-14T17:39:35.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:34 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.96:0/3123834103' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:39:35.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:34 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:35.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:34 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:35.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:34 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:35.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:34 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:35.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:34 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:35.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:34 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:35.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:34 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:35.979 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:35 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:35.629+0000 7f9ea15a4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:39:35.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:35 smithi040 ceph-mon[27571]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:35.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:35 smithi040 ceph-mon[27571]: mgrmap e14: y(active, since 82s), standbys: z 2024-02-14T17:39:35.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:35 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-14T17:39:35.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:35 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/504730726' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90eb8397-c65b-458e-a9fe-16e8b137a2ea"}]: dispatch 2024-02-14T17:39:35.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:35 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/504730726' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90eb8397-c65b-458e-a9fe-16e8b137a2ea"}]': finished 2024-02-14T17:39:35.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:35 smithi040 ceph-mon[27571]: osdmap e5: 1 total, 0 up, 1 in 2024-02-14T17:39:35.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:35 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:36.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:35 smithi027 ceph-mon[25828]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:36.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:35 smithi027 ceph-mon[25828]: mgrmap e14: y(active, since 82s), standbys: z 2024-02-14T17:39:36.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:35 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-14T17:39:36.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:35 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/504730726' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90eb8397-c65b-458e-a9fe-16e8b137a2ea"}]: dispatch 2024-02-14T17:39:36.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:35 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/504730726' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90eb8397-c65b-458e-a9fe-16e8b137a2ea"}]': finished 2024-02-14T17:39:36.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:35 smithi027 ceph-mon[25828]: osdmap e5: 1 total, 0 up, 1 in 2024-02-14T17:39:36.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:35 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:36.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:35 smithi096 ceph-mon[27796]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:36.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:35 smithi096 ceph-mon[27796]: mgrmap e14: y(active, since 82s), standbys: z 2024-02-14T17:39:36.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:35 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-14T17:39:36.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:35 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/504730726' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90eb8397-c65b-458e-a9fe-16e8b137a2ea"}]: dispatch 2024-02-14T17:39:36.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:35 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/504730726' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90eb8397-c65b-458e-a9fe-16e8b137a2ea"}]': finished 2024-02-14T17:39:36.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:35 smithi096 ceph-mon[27796]: osdmap e5: 1 total, 0 up, 1 in 2024-02-14T17:39:36.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:35 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:36.478 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:36 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:36.209+0000 7f9ea15a4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:39:36.479 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:36 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:36.308+0000 7f9ea15a4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:39:36.876 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:39:36 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:39:36.533+0000 7f9ea15a4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:39:37.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:36 smithi040 ceph-mon[27571]: Standby manager daemon x started 2024-02-14T17:39:37.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:36 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.40:0/4051296347' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:39:37.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:36 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.40:0/4051296347' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:39:37.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:36 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.40:0/4051296347' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:39:37.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:36 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.40:0/4051296347' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:39:37.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:36 smithi027 ceph-mon[25828]: Standby manager daemon x started 2024-02-14T17:39:37.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:36 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.40:0/4051296347' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:39:37.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:36 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.40:0/4051296347' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:39:37.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:36 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.40:0/4051296347' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:39:37.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:36 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.40:0/4051296347' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:39:37.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:36 smithi096 ceph-mon[27796]: Standby manager daemon x started 2024-02-14T17:39:37.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:36 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.40:0/4051296347' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:39:37.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:36 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.40:0/4051296347' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:39:37.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:36 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.40:0/4051296347' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:39:37.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:36 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.40:0/4051296347' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:39:38.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:37 smithi040 ceph-mon[27571]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:38.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:37 smithi040 ceph-mon[27571]: mgrmap e15: y(active, since 84s), standbys: x, z 2024-02-14T17:39:38.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:37 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:39:38.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:37 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/2989352595' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:39:38.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:37 smithi027 ceph-mon[25828]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:38.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:37 smithi027 ceph-mon[25828]: mgrmap e15: y(active, since 84s), standbys: x, z 2024-02-14T17:39:38.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:37 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:39:38.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:37 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/2989352595' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:39:38.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:37 smithi096 ceph-mon[27796]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:38.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:37 smithi096 ceph-mon[27796]: mgrmap e15: y(active, since 84s), standbys: x, z 2024-02-14T17:39:38.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:37 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:39:38.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:37 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/2989352595' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:39:40.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:39 smithi027 ceph-mon[25828]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:40.228 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:39 smithi040 ceph-mon[27571]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:40.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:39 smithi096 ceph-mon[27796]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:42.135 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:41 smithi027 ceph-mon[25828]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:42.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:41 smithi040 ceph-mon[27571]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:42.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:41 smithi096 ceph-mon[27796]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:44.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:43 smithi027 ceph-mon[25828]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:44.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:43 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-14T17:39:44.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:43 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:44.228 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:43 smithi040 ceph-mon[27571]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:44.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:43 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-14T17:39:44.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:43 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:44.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:43 smithi096 ceph-mon[27796]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:44.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:43 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-14T17:39:44.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:43 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:45.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:44 smithi040 ceph-mon[27571]: Deploying daemon osd.0 on smithi027 2024-02-14T17:39:45.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:44 smithi027 ceph-mon[25828]: Deploying daemon osd.0 on smithi027 2024-02-14T17:39:45.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:44 smithi096 ceph-mon[27796]: Deploying daemon osd.0 on smithi027 2024-02-14T17:39:46.135 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:45 smithi027 ceph-mon[25828]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:46.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:45 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:46.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:45 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:46.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:45 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:46.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:45 smithi040 ceph-mon[27571]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:46.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:45 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:46.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:45 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:46.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:45 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:46.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:45 smithi096 ceph-mon[27796]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:46.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:45 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:46.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:45 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:46.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:45 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:48.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:47 smithi027 ceph-mon[25828]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:48.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:47 smithi040 ceph-mon[27571]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:48.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:47 smithi096 ceph-mon[27796]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:50.151 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:49 smithi096 ceph-mon[27796]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:50.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:49 smithi040 ceph-mon[27571]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:50.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:49 smithi027 ceph-mon[25828]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:51.933 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:51 smithi027 ceph-mon[25828]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:51.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:51 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:51.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:51 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:51.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:51 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:51.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:51 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:51.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:51 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:51.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:51 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:51.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:51 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:51.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:51 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:51.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:51 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:52.145 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:51 smithi040 ceph-mon[27571]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:52.145 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:51 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:52.145 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:51 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:52.146 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:51 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:52.146 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:51 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:52.146 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:51 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:52.146 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:51 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:52.146 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:51 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:52.146 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:51 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:52.146 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:51 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:52.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:51 smithi096 ceph-mon[27796]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:52.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:51 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:52.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:51 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:52.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:51 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:52.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:51 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:52.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:51 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:52.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:51 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:52.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:51 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:52.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:51 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:52.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:51 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:53.923 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:53 smithi027 ceph-mon[25828]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:53.923 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:53 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:53.923 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:53 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:53.923 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:53 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:53.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:53 smithi040 ceph-mon[27571]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:53.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:53 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:53.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:53 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:53.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:53 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:54.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:53 smithi096 ceph-mon[27796]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:54.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:53 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:54.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:53 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:54.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:53 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:54.728 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 0 on host 'smithi027' 2024-02-14T17:39:54.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:54 smithi040 ceph-mon[27571]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-14T17:39:54.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:54 smithi027 ceph-mon[25828]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-14T17:39:55.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:54 smithi096 ceph-mon[27796]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-14T17:39:55.370 DEBUG:teuthology.orchestra.run.smithi027:osd.0> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@osd.0.service 2024-02-14T17:39:55.372 INFO:tasks.cephadm:Deploying osd.1 on smithi040 with /dev/nvme4n1... 2024-02-14T17:39:55.372 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-02-14T17:39:55.625 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.b/config 2024-02-14T17:39:55.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:55.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-14T17:39:55.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: osdmap e6: 1 total, 0 up, 1 in 2024-02-14T17:39:55.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-02-14T17:39:55.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:55.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:55.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:55.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:55.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:39:55.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: Adjusting osd_memory_target on smithi027 to 16064M 2024-02-14T17:39:55.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:55.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:55.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:55.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:55 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:55.936 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:55.936 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-14T17:39:55.937 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: osdmap e6: 1 total, 0 up, 1 in 2024-02-14T17:39:55.937 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-02-14T17:39:55.937 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:55.937 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:55.937 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:55.937 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:55.937 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:39:55.937 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: Adjusting osd_memory_target on smithi027 to 16064M 2024-02-14T17:39:55.937 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:55.937 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:55.937 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:55.937 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:55 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:55.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:55.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-14T17:39:55.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: osdmap e6: 1 total, 0 up, 1 in 2024-02-14T17:39:55.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-02-14T17:39:55.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:55.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:55.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:55.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:55.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:39:55.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: Adjusting osd_memory_target on smithi027 to 16064M 2024-02-14T17:39:55.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:55.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:55.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:55.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:39:56.483 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:39:56 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0[38251]: 2024-02-14T17:39:56.120+0000 7f9926d16700 -1 osd.0 0 waiting for initial osdmap 2024-02-14T17:39:56.483 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:39:56 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0[38251]: 2024-02-14T17:39:56.123+0000 7f991eb01700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:39:56.905 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:56 smithi027 ceph-mon[25828]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-02-14T17:39:56.905 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:56 smithi027 ceph-mon[25828]: osdmap e7: 1 total, 0 up, 1 in 2024-02-14T17:39:56.905 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:56 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:56.906 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:56 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:56.906 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:56 smithi027 ceph-mon[25828]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' 2024-02-14T17:39:56.906 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:56 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:56.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:56 smithi040 ceph-mon[27571]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-02-14T17:39:56.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:56 smithi040 ceph-mon[27571]: osdmap e7: 1 total, 0 up, 1 in 2024-02-14T17:39:56.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:56 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:56.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:56 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:56.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:56 smithi040 ceph-mon[27571]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' 2024-02-14T17:39:56.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:56 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:57.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:56 smithi096 ceph-mon[27796]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-02-14T17:39:57.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:56 smithi096 ceph-mon[27796]: osdmap e7: 1 total, 0 up, 1 in 2024-02-14T17:39:57.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:56 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:57.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:56 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:57.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:56 smithi096 ceph-mon[27796]: from='osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731]' entity='osd.0' 2024-02-14T17:39:57.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:56 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/nvme4n1 2024-02-14T17:39:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-14T17:39:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-02-14T17:39:57.567 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-02-14T17:39:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-02-14T17:39:57.567 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012319 s, 851 MB/s 2024-02-14T17:39:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-02-14T17:39:58.151 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:58 smithi040 ceph-mon[27571]: purged_snaps scrub starts 2024-02-14T17:39:58.151 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:58 smithi040 ceph-mon[27571]: purged_snaps scrub ok 2024-02-14T17:39:58.151 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:58 smithi040 ceph-mon[27571]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:58.151 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:58 smithi040 ceph-mon[27571]: osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731] boot 2024-02-14T17:39:58.151 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:58 smithi040 ceph-mon[27571]: osdmap e8: 1 total, 1 up, 1 in 2024-02-14T17:39:58.151 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:58 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:58.180 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi040:/dev/nvme4n1 2024-02-14T17:39:58.420 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.b/config 2024-02-14T17:39:58.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:58 smithi027 ceph-mon[25828]: purged_snaps scrub starts 2024-02-14T17:39:58.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:58 smithi027 ceph-mon[25828]: purged_snaps scrub ok 2024-02-14T17:39:58.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:58 smithi027 ceph-mon[25828]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:58.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:58 smithi027 ceph-mon[25828]: osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731] boot 2024-02-14T17:39:58.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:58 smithi027 ceph-mon[25828]: osdmap e8: 1 total, 1 up, 1 in 2024-02-14T17:39:58.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:58 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:58.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:58 smithi096 ceph-mon[27796]: purged_snaps scrub starts 2024-02-14T17:39:58.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:58 smithi096 ceph-mon[27796]: purged_snaps scrub ok 2024-02-14T17:39:58.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:58 smithi096 ceph-mon[27796]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:39:58.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:58 smithi096 ceph-mon[27796]: osd.0 [v2:172.21.15.27:6802/1313296731,v1:172.21.15.27:6803/1313296731] boot 2024-02-14T17:39:58.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:58 smithi096 ceph-mon[27796]: osdmap e8: 1 total, 1 up, 1 in 2024-02-14T17:39:58.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:58 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:39:59.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:39:59 smithi040 ceph-mon[27571]: osdmap e9: 1 total, 1 up, 1 in 2024-02-14T17:39:59.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:39:59 smithi027 ceph-mon[25828]: osdmap e9: 1 total, 1 up, 1 in 2024-02-14T17:39:59.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:39:59 smithi096 ceph-mon[27796]: osdmap e9: 1 total, 1 up, 1 in 2024-02-14T17:40:00.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:00 smithi040 ceph-mon[27571]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:00.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:00 smithi040 ceph-mon[27571]: overall HEALTH_OK 2024-02-14T17:40:00.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:00 smithi027 ceph-mon[25828]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:00.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:00 smithi027 ceph-mon[25828]: overall HEALTH_OK 2024-02-14T17:40:00.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:00 smithi096 ceph-mon[27796]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:00.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:00 smithi096 ceph-mon[27796]: overall HEALTH_OK 2024-02-14T17:40:01.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:01 smithi027 ceph-mon[25828]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:01.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:01 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:40:01.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:01 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:40:01.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:01 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:01.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:01 smithi040 ceph-mon[27571]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:01.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:01 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:40:01.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:01 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:40:01.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:01 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:01.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:01 smithi096 ceph-mon[27796]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:01.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:40:01.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:40:01.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:02.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:02 smithi040 ceph-mon[27571]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:02.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:02 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:02.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:02 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:02.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:02 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:02.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:02 smithi027 ceph-mon[25828]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:02.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:02 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:02.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:02 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:02.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:02 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:02.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:02 smithi096 ceph-mon[27796]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:02.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:02 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:02.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:02 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:02.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:02 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:03.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:03 smithi040 ceph-mon[27571]: Detected new or changed devices on smithi027 2024-02-14T17:40:03.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:03 smithi040 ceph-mon[27571]: from='client.? 172.21.15.40:0/2395791720' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a33d7701-d5f3-482e-8e5a-d658508b5468"}]: dispatch 2024-02-14T17:40:03.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:03 smithi040 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a33d7701-d5f3-482e-8e5a-d658508b5468"}]: dispatch 2024-02-14T17:40:03.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:03 smithi040 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a33d7701-d5f3-482e-8e5a-d658508b5468"}]': finished 2024-02-14T17:40:03.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:03 smithi040 ceph-mon[27571]: osdmap e10: 2 total, 1 up, 2 in 2024-02-14T17:40:03.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:03 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:03.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:03 smithi027 ceph-mon[25828]: Detected new or changed devices on smithi027 2024-02-14T17:40:03.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:03 smithi027 ceph-mon[25828]: from='client.? 172.21.15.40:0/2395791720' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a33d7701-d5f3-482e-8e5a-d658508b5468"}]: dispatch 2024-02-14T17:40:03.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:03 smithi027 ceph-mon[25828]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a33d7701-d5f3-482e-8e5a-d658508b5468"}]: dispatch 2024-02-14T17:40:03.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:03 smithi027 ceph-mon[25828]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a33d7701-d5f3-482e-8e5a-d658508b5468"}]': finished 2024-02-14T17:40:03.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:03 smithi027 ceph-mon[25828]: osdmap e10: 2 total, 1 up, 2 in 2024-02-14T17:40:03.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:03 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:03.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:03 smithi096 ceph-mon[27796]: Detected new or changed devices on smithi027 2024-02-14T17:40:03.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:03 smithi096 ceph-mon[27796]: from='client.? 172.21.15.40:0/2395791720' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a33d7701-d5f3-482e-8e5a-d658508b5468"}]: dispatch 2024-02-14T17:40:03.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:03 smithi096 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a33d7701-d5f3-482e-8e5a-d658508b5468"}]: dispatch 2024-02-14T17:40:03.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:03 smithi096 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a33d7701-d5f3-482e-8e5a-d658508b5468"}]': finished 2024-02-14T17:40:03.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:03 smithi096 ceph-mon[27796]: osdmap e10: 2 total, 1 up, 2 in 2024-02-14T17:40:03.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:03 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:04.264 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:04 smithi027 ceph-mon[25828]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:04.264 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:04 smithi027 ceph-mon[25828]: from='client.? 172.21.15.40:0/527223349' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:40:04.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:04 smithi040 ceph-mon[27571]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:04.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:04 smithi040 ceph-mon[27571]: from='client.? 172.21.15.40:0/527223349' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:40:04.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:04 smithi096 ceph-mon[27796]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:04.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:04 smithi096 ceph-mon[27796]: from='client.? 172.21.15.40:0/527223349' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:40:06.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:06 smithi040 ceph-mon[27571]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:06.440 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:06 smithi027 ceph-mon[25828]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:06.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:06 smithi096 ceph-mon[27796]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:08.228 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:08 smithi040 ceph-mon[27571]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:08.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:08 smithi027 ceph-mon[25828]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:08.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:08 smithi096 ceph-mon[27796]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:10.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:10 smithi040 ceph-mon[27571]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:10.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:10 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-14T17:40:10.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:10 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:10.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:10 smithi027 ceph-mon[25828]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:10.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:10 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-14T17:40:10.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:10 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:10.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:10 smithi096 ceph-mon[27796]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:10.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:10 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-14T17:40:10.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:10 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:11.270 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:11 smithi040 ceph-mon[27571]: Deploying daemon osd.1 on smithi040 2024-02-14T17:40:11.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:11 smithi096 ceph-mon[27796]: Deploying daemon osd.1 on smithi040 2024-02-14T17:40:11.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:11 smithi027 ceph-mon[25828]: Deploying daemon osd.1 on smithi040 2024-02-14T17:40:12.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:12 smithi040 ceph-mon[27571]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:12.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:12 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:12.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:12 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:12.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:12 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:12.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:12 smithi027 ceph-mon[25828]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:12.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:12.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:12.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:12.496 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:12 smithi096 ceph-mon[27796]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:12.496 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:12 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:12.497 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:12 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:12.497 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:12 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:13.913 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:13 smithi040 ceph-mon[27571]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:13.913 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:13 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:13.913 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:13 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:13.913 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:13 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:14.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:13 smithi096 ceph-mon[27796]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:14.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:13 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:14.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:13 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:14.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:13 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:14.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:13 smithi027 ceph-mon[25828]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:14.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:13 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:14.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:13 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:14.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:13 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:15.864 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:15 smithi040 ceph-mon[27571]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:16.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:15 smithi027 ceph-mon[25828]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:16.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:15 smithi096 ceph-mon[27796]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:17.447 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:17 smithi040 ceph-mon[27571]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:17.447 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:17 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:17.447 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:17 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:17.448 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:17 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:17.448 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:17 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:17.448 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:17 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:17.448 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:17 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:17.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:17 smithi027 ceph-mon[25828]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:17.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:17.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:17.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:17.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:17.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:17.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:17 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:17.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:17 smithi096 ceph-mon[27796]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:17.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:17.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:17.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:17.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:17.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:17.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:17 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:19.728 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:19 smithi040 ceph-mon[27571]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:19.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:19 smithi027 ceph-mon[25828]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:19.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:19 smithi096 ceph-mon[27796]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:20.183 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 1 on host 'smithi040' 2024-02-14T17:40:20.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:20 smithi040 ceph-mon[27571]: from='osd.1 [v2:172.21.15.40:6800/4075500575,v1:172.21.15.40:6801/4075500575]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:40:20.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:20 smithi040 ceph-mon[27571]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:40:20.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:20 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:20.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:20 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:20.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:20 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:20.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:20 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:40:20.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:20 smithi040 ceph-mon[27571]: Adjusting osd_memory_target on smithi040 to 16069M 2024-02-14T17:40:20.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:20 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:20.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:20 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:20.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:20 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:20.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:20 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:20 smithi027 ceph-mon[25828]: from='osd.1 [v2:172.21.15.40:6800/4075500575,v1:172.21.15.40:6801/4075500575]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:40:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:20 smithi027 ceph-mon[25828]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:40:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:20 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:20 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:20 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:20 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:40:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:20 smithi027 ceph-mon[25828]: Adjusting osd_memory_target on smithi040 to 16069M 2024-02-14T17:40:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:20 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:20 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:20.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:20 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:20.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:20 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:20.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:20 smithi096 ceph-mon[27796]: from='osd.1 [v2:172.21.15.40:6800/4075500575,v1:172.21.15.40:6801/4075500575]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:40:20.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:20 smithi096 ceph-mon[27796]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:40:20.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:20 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:20.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:20 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:20.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:20 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:20.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:20 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:40:20.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:20 smithi096 ceph-mon[27796]: Adjusting osd_memory_target on smithi040 to 16069M 2024-02-14T17:40:20.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:20 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:20.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:20 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:20.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:20 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:20.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:20 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:20.860 DEBUG:teuthology.orchestra.run.smithi040:osd.1> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@osd.1.service 2024-02-14T17:40:20.863 INFO:tasks.cephadm:Deploying osd.2 on smithi096 with /dev/nvme4n1... 2024-02-14T17:40:20.863 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-02-14T17:40:21.110 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.c/config 2024-02-14T17:40:21.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:21 smithi096 ceph-mon[27796]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:21.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:21 smithi096 ceph-mon[27796]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-14T17:40:21.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:21 smithi096 ceph-mon[27796]: osdmap e11: 2 total, 1 up, 2 in 2024-02-14T17:40:21.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:21 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:21.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:21 smithi096 ceph-mon[27796]: from='osd.1 [v2:172.21.15.40:6800/4075500575,v1:172.21.15.40:6801/4075500575]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-02-14T17:40:21.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:21 smithi096 ceph-mon[27796]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-02-14T17:40:21.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:21 smithi040 ceph-mon[27571]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:21.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:21 smithi040 ceph-mon[27571]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-14T17:40:21.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:21 smithi040 ceph-mon[27571]: osdmap e11: 2 total, 1 up, 2 in 2024-02-14T17:40:21.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:21 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:21.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:21 smithi040 ceph-mon[27571]: from='osd.1 [v2:172.21.15.40:6800/4075500575,v1:172.21.15.40:6801/4075500575]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-02-14T17:40:21.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:21 smithi040 ceph-mon[27571]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-02-14T17:40:21.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:21 smithi027 ceph-mon[25828]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:21.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:21 smithi027 ceph-mon[25828]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-14T17:40:21.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:21 smithi027 ceph-mon[25828]: osdmap e11: 2 total, 1 up, 2 in 2024-02-14T17:40:21.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:21 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:21.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:21 smithi027 ceph-mon[25828]: from='osd.1 [v2:172.21.15.40:6800/4075500575,v1:172.21.15.40:6801/4075500575]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-02-14T17:40:21.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:21 smithi027 ceph-mon[25828]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-02-14T17:40:22.229 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:40:21 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1[32850]: 2024-02-14T17:40:21.880+0000 7feb9b004700 -1 osd.1 0 waiting for initial osdmap 2024-02-14T17:40:22.229 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:40:21 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1[32850]: 2024-02-14T17:40:21.888+0000 7feb92def700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:40:22.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:22 smithi040 ceph-mon[27571]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-02-14T17:40:22.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:22 smithi040 ceph-mon[27571]: osdmap e12: 2 total, 1 up, 2 in 2024-02-14T17:40:22.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:22 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:22.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:22 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:22.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:22 smithi040 ceph-mon[27571]: from='osd.1 ' entity='osd.1' 2024-02-14T17:40:22.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:22 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:22.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:22 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:22.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:22 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:22.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:22 smithi027 ceph-mon[25828]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-02-14T17:40:22.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:22 smithi027 ceph-mon[25828]: osdmap e12: 2 total, 1 up, 2 in 2024-02-14T17:40:22.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:22.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:22.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:22 smithi027 ceph-mon[25828]: from='osd.1 ' entity='osd.1' 2024-02-14T17:40:22.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:22.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:22.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:22 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:22.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:22 smithi096 ceph-mon[27796]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-02-14T17:40:22.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:22 smithi096 ceph-mon[27796]: osdmap e12: 2 total, 1 up, 2 in 2024-02-14T17:40:22.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:22 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:22.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:22 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:22.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:22 smithi096 ceph-mon[27796]: from='osd.1 ' entity='osd.1' 2024-02-14T17:40:22.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:22 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:22.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:22 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:22.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:22 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:22.932 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/nvme4n1 2024-02-14T17:40:22.932 INFO:teuthology.orchestra.run.smithi096.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-14T17:40:22.932 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-02-14T17:40:22.932 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-02-14T17:40:22.932 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-02-14T17:40:22.932 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125574 s, 835 MB/s 2024-02-14T17:40:22.932 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-02-14T17:40:23.593 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:23 smithi096 ceph-mon[27796]: purged_snaps scrub starts 2024-02-14T17:40:23.593 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:23 smithi096 ceph-mon[27796]: purged_snaps scrub ok 2024-02-14T17:40:23.593 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:23 smithi096 ceph-mon[27796]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:23.593 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:23 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:23.593 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:23 smithi096 ceph-mon[27796]: osd.1 [v2:172.21.15.40:6800/4075500575,v1:172.21.15.40:6801/4075500575] boot 2024-02-14T17:40:23.593 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:23 smithi096 ceph-mon[27796]: osdmap e13: 2 total, 2 up, 2 in 2024-02-14T17:40:23.593 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:23 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:23.620 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi096:/dev/nvme4n1 2024-02-14T17:40:23.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:23 smithi040 ceph-mon[27571]: purged_snaps scrub starts 2024-02-14T17:40:23.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:23 smithi040 ceph-mon[27571]: purged_snaps scrub ok 2024-02-14T17:40:23.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:23 smithi040 ceph-mon[27571]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:23.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:23 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:23.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:23 smithi040 ceph-mon[27571]: osd.1 [v2:172.21.15.40:6800/4075500575,v1:172.21.15.40:6801/4075500575] boot 2024-02-14T17:40:23.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:23 smithi040 ceph-mon[27571]: osdmap e13: 2 total, 2 up, 2 in 2024-02-14T17:40:23.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:23 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:23.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:23 smithi027 ceph-mon[25828]: purged_snaps scrub starts 2024-02-14T17:40:23.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:23 smithi027 ceph-mon[25828]: purged_snaps scrub ok 2024-02-14T17:40:23.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:23 smithi027 ceph-mon[25828]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:40:23.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:23 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:23.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:23 smithi027 ceph-mon[25828]: osd.1 [v2:172.21.15.40:6800/4075500575,v1:172.21.15.40:6801/4075500575] boot 2024-02-14T17:40:23.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:23 smithi027 ceph-mon[25828]: osdmap e13: 2 total, 2 up, 2 in 2024-02-14T17:40:23.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:23 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:40:23.866 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.c/config 2024-02-14T17:40:25.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:25 smithi040 ceph-mon[27571]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:25.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:25 smithi040 ceph-mon[27571]: osdmap e14: 2 total, 2 up, 2 in 2024-02-14T17:40:25.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:25 smithi027 ceph-mon[25828]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:25.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:25 smithi027 ceph-mon[25828]: osdmap e14: 2 total, 2 up, 2 in 2024-02-14T17:40:25.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:25 smithi096 ceph-mon[27796]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:25.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:25 smithi096 ceph-mon[27796]: osdmap e14: 2 total, 2 up, 2 in 2024-02-14T17:40:26.703 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:26 smithi040 ceph-mon[27571]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:26.703 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:26 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:40:26.703 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:26 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:40:26.703 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:26 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:26.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:26 smithi027 ceph-mon[25828]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:26.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:26 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:40:26.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:26 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:40:26.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:26 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:26.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:26 smithi096 ceph-mon[27796]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:26.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:26 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:40:26.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:26 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:40:26.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:26 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:27.728 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:27 smithi040 ceph-mon[27571]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:27.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:27 smithi027 ceph-mon[25828]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:27.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:27 smithi096 ceph-mon[27796]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:28.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:28 smithi040 ceph-mon[27571]: from='client.? 172.21.15.96:0/1432728781' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eac09c38-7d9b-43e1-8e02-7b99103304c7"}]: dispatch 2024-02-14T17:40:28.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:28 smithi040 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eac09c38-7d9b-43e1-8e02-7b99103304c7"}]: dispatch 2024-02-14T17:40:28.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:28 smithi040 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eac09c38-7d9b-43e1-8e02-7b99103304c7"}]': finished 2024-02-14T17:40:28.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:28 smithi040 ceph-mon[27571]: osdmap e15: 3 total, 2 up, 3 in 2024-02-14T17:40:28.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:28 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:28.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:28 smithi027 ceph-mon[25828]: from='client.? 172.21.15.96:0/1432728781' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eac09c38-7d9b-43e1-8e02-7b99103304c7"}]: dispatch 2024-02-14T17:40:28.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:28 smithi027 ceph-mon[25828]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eac09c38-7d9b-43e1-8e02-7b99103304c7"}]: dispatch 2024-02-14T17:40:28.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:28 smithi027 ceph-mon[25828]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eac09c38-7d9b-43e1-8e02-7b99103304c7"}]': finished 2024-02-14T17:40:28.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:28 smithi027 ceph-mon[25828]: osdmap e15: 3 total, 2 up, 3 in 2024-02-14T17:40:28.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:28 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:28.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:28 smithi096 ceph-mon[27796]: from='client.? 172.21.15.96:0/1432728781' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eac09c38-7d9b-43e1-8e02-7b99103304c7"}]: dispatch 2024-02-14T17:40:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:28 smithi096 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eac09c38-7d9b-43e1-8e02-7b99103304c7"}]: dispatch 2024-02-14T17:40:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:28 smithi096 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eac09c38-7d9b-43e1-8e02-7b99103304c7"}]': finished 2024-02-14T17:40:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:28 smithi096 ceph-mon[27796]: osdmap e15: 3 total, 2 up, 3 in 2024-02-14T17:40:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:28 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:29.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:29 smithi040 ceph-mon[27571]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:29.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:29 smithi027 ceph-mon[25828]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:29.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:29 smithi096 ceph-mon[27796]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:30.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:30 smithi040 ceph-mon[27571]: from='client.? 172.21.15.96:0/941483485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:40:30.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:30 smithi027 ceph-mon[25828]: from='client.? 172.21.15.96:0/941483485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:40:30.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:30 smithi096 ceph-mon[27796]: from='client.? 172.21.15.96:0/941483485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:40:31.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:31 smithi040 ceph-mon[27571]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:31.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:31 smithi027 ceph-mon[25828]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:31.772 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:31 smithi096 ceph-mon[27796]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:32.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:32 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:32 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:32 smithi096 ceph-mon[27796]: Detected new or changed devices on smithi040 2024-02-14T17:40:32.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:32 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:32 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:32 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:32 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:32 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:32 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:32 smithi040 ceph-mon[27571]: Detected new or changed devices on smithi040 2024-02-14T17:40:32.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:32 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:32 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:32 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:32 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:32 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:32 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:32 smithi027 ceph-mon[25828]: Detected new or changed devices on smithi040 2024-02-14T17:40:32.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:32 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:32 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:32 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:32.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:32 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:33.978 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:33 smithi040 ceph-mon[27571]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:33.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:33 smithi027 ceph-mon[25828]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:34.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:33 smithi096 ceph-mon[27796]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:35.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:35 smithi096 ceph-mon[27796]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:35.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:35 smithi040 ceph-mon[27571]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:35.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:35 smithi027 ceph-mon[25828]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:36.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:36 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-14T17:40:36.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:36 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:36.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:36 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-14T17:40:36.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:36 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:36.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:36 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-14T17:40:36.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:36 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:37.749 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:37 smithi096 ceph-mon[27796]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:37.750 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:37 smithi096 ceph-mon[27796]: Deploying daemon osd.2 on smithi096 2024-02-14T17:40:37.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:37 smithi040 ceph-mon[27571]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:37.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:37 smithi040 ceph-mon[27571]: Deploying daemon osd.2 on smithi096 2024-02-14T17:40:37.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:37 smithi027 ceph-mon[25828]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:37.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:37 smithi027 ceph-mon[25828]: Deploying daemon osd.2 on smithi096 2024-02-14T17:40:39.782 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:39 smithi096 ceph-mon[27796]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:39.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:39 smithi040 ceph-mon[27571]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:39.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:39 smithi027 ceph-mon[25828]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:41.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:41 smithi027 ceph-mon[25828]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:41.978 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:41 smithi040 ceph-mon[27571]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:42.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:41 smithi096 ceph-mon[27796]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:43.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:42 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:42 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:42 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:43.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:42 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:43.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:42 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:43.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:42 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.173 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:42 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:42 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.174 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:42 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:42 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:42 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:42 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:43.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:42 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:43.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:42 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:43.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:42 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:42 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:42 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:42 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:42 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:42 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:42 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:43.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:42 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:43.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:42 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:43.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:42 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:42 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:42 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:43.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:42 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:44.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:44 smithi027 ceph-mon[25828]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:44.385 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:44 smithi096 ceph-mon[27796]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:44.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:44 smithi040 ceph-mon[27571]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:44.990 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 2 on host 'smithi096' 2024-02-14T17:40:45.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:45 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:45.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:45 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:45.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:45 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:40:45.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:45 smithi096 ceph-mon[27796]: from='osd.2 [v2:172.21.15.96:6800/1178359084,v1:172.21.15.96:6801/1178359084]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-14T17:40:45.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:45 smithi096 ceph-mon[27796]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-14T17:40:45.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:45 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:45.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:45 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:45.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:45 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:45.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:45 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:45.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:45 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:45.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:45 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:45.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:45 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:45.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:45 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:40:45.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:45 smithi040 ceph-mon[27571]: from='osd.2 [v2:172.21.15.96:6800/1178359084,v1:172.21.15.96:6801/1178359084]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-14T17:40:45.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:45 smithi040 ceph-mon[27571]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-14T17:40:45.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:45 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:45.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:45 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:45.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:45 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:45.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:45 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:45.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:45 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:45 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:45 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:45 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:40:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:45 smithi027 ceph-mon[25828]: from='osd.2 [v2:172.21.15.96:6800/1178359084,v1:172.21.15.96:6801/1178359084]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-14T17:40:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:45 smithi027 ceph-mon[25828]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-14T17:40:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:45 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:45 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:45.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:45 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:45.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:45 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:45.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:45 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:45.667 DEBUG:teuthology.orchestra.run.smithi096:osd.2> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@osd.2.service 2024-02-14T17:40:45.669 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-02-14T17:40:45.670 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph osd stat -f json 2024-02-14T17:40:46.378 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:46 smithi027 ceph-mon[25828]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:46.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:46 smithi027 ceph-mon[25828]: Adjusting osd_memory_target on smithi096 to 16064M 2024-02-14T17:40:46.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:46 smithi027 ceph-mon[25828]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-14T17:40:46.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:46 smithi027 ceph-mon[25828]: osdmap e16: 3 total, 2 up, 3 in 2024-02-14T17:40:46.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:46 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:46.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:46 smithi027 ceph-mon[25828]: from='osd.2 [v2:172.21.15.96:6800/1178359084,v1:172.21.15.96:6801/1178359084]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-02-14T17:40:46.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:46 smithi027 ceph-mon[25828]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-02-14T17:40:46.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:46 smithi040 ceph-mon[27571]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:46.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:46 smithi040 ceph-mon[27571]: Adjusting osd_memory_target on smithi096 to 16064M 2024-02-14T17:40:46.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:46 smithi040 ceph-mon[27571]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-14T17:40:46.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:46 smithi040 ceph-mon[27571]: osdmap e16: 3 total, 2 up, 3 in 2024-02-14T17:40:46.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:46 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:46.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:46 smithi040 ceph-mon[27571]: from='osd.2 [v2:172.21.15.96:6800/1178359084,v1:172.21.15.96:6801/1178359084]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-02-14T17:40:46.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:46 smithi040 ceph-mon[27571]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-02-14T17:40:46.497 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:40:46.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:46 smithi096 ceph-mon[27796]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:46.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:46 smithi096 ceph-mon[27796]: Adjusting osd_memory_target on smithi096 to 16064M 2024-02-14T17:40:46.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:46 smithi096 ceph-mon[27796]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-14T17:40:46.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:46 smithi096 ceph-mon[27796]: osdmap e16: 3 total, 2 up, 3 in 2024-02-14T17:40:46.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:46 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:46.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:46 smithi096 ceph-mon[27796]: from='osd.2 [v2:172.21.15.96:6800/1178359084,v1:172.21.15.96:6801/1178359084]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-02-14T17:40:46.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:46 smithi096 ceph-mon[27796]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-02-14T17:40:47.296 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:40:46 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2[34218]: 2024-02-14T17:40:46.980+0000 7f2120f8f700 -1 osd.2 0 waiting for initial osdmap 2024-02-14T17:40:47.296 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:40:46 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2[34218]: 2024-02-14T17:40:46.984+0000 7f211cd82700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:40:48.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:47 smithi027 ceph-mon[25828]: purged_snaps scrub starts 2024-02-14T17:40:48.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:47 smithi027 ceph-mon[25828]: purged_snaps scrub ok 2024-02-14T17:40:48.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:47 smithi027 ceph-mon[25828]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:48.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:47 smithi027 ceph-mon[25828]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-02-14T17:40:48.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:47 smithi027 ceph-mon[25828]: osdmap e17: 3 total, 2 up, 3 in 2024-02-14T17:40:48.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:47 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:48.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:47 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:48.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:47 smithi096 ceph-mon[27796]: purged_snaps scrub starts 2024-02-14T17:40:48.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:47 smithi096 ceph-mon[27796]: purged_snaps scrub ok 2024-02-14T17:40:48.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:47 smithi096 ceph-mon[27796]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:48.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:47 smithi096 ceph-mon[27796]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-02-14T17:40:48.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:47 smithi096 ceph-mon[27796]: osdmap e17: 3 total, 2 up, 3 in 2024-02-14T17:40:48.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:47 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:48.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:47 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:48.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:47 smithi040 ceph-mon[27571]: purged_snaps scrub starts 2024-02-14T17:40:48.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:47 smithi040 ceph-mon[27571]: purged_snaps scrub ok 2024-02-14T17:40:48.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:47 smithi040 ceph-mon[27571]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:40:48.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:47 smithi040 ceph-mon[27571]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-02-14T17:40:48.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:47 smithi040 ceph-mon[27571]: osdmap e17: 3 total, 2 up, 3 in 2024-02-14T17:40:48.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:47 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:48.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:47 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:40:49.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:48 smithi027 ceph-mon[25828]: osd.2 [v2:172.21.15.96:6800/1178359084,v1:172.21.15.96:6801/1178359084] boot 2024-02-14T17:40:49.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:48 smithi027 ceph-mon[25828]: osdmap e18: 3 total, 3 up, 3 in 2024-02-14T17:40:49.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:48 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:49.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:48 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-14T17:40:49.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:48 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/4053853637' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-14T17:40:49.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:48 smithi096 ceph-mon[27796]: osd.2 [v2:172.21.15.96:6800/1178359084,v1:172.21.15.96:6801/1178359084] boot 2024-02-14T17:40:49.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:48 smithi096 ceph-mon[27796]: osdmap e18: 3 total, 3 up, 3 in 2024-02-14T17:40:49.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:48 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:49.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:48 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-14T17:40:49.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:48 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/4053853637' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-14T17:40:49.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:48 smithi040 ceph-mon[27571]: osd.2 [v2:172.21.15.96:6800/1178359084,v1:172.21.15.96:6801/1178359084] boot 2024-02-14T17:40:49.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:48 smithi040 ceph-mon[27571]: osdmap e18: 3 total, 3 up, 3 in 2024-02-14T17:40:49.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:48 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:40:49.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:48 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-14T17:40:49.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:48 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/4053853637' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-14T17:40:49.587 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":18,"num_osds":3,"num_up_osds":3,"osd_up_since":1707932447,"num_in_osds":3,"osd_in_since":1707932428,"num_remapped_pgs":0} 2024-02-14T17:40:49.587 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-14T17:40:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:40:50.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:49 smithi027 ceph-mon[25828]: pgmap v86: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:50.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:49 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-14T17:40:50.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:49 smithi027 ceph-mon[25828]: osdmap e19: 3 total, 3 up, 3 in 2024-02-14T17:40:50.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:49 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-14T17:40:50.276 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:49 smithi096 ceph-mon[27796]: pgmap v86: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:50.276 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:49 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-14T17:40:50.276 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:49 smithi096 ceph-mon[27796]: osdmap e19: 3 total, 3 up, 3 in 2024-02-14T17:40:50.277 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:49 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-14T17:40:50.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:49 smithi040 ceph-mon[27571]: pgmap v86: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:50.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:49 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-14T17:40:50.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:49 smithi040 ceph-mon[27571]: osdmap e19: 3 total, 3 up, 3 in 2024-02-14T17:40:50.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:49 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-14T17:40:50.983 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:40:50 smithi027 sudo[40508]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme4n1 2024-02-14T17:40:50.983 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:40:50 smithi027 sudo[40508]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-14T17:40:50.983 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:40:50 smithi027 sudo[40508]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:40:50.983 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:40:50 smithi027 sudo[40508]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:40:51.281 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:50 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-14T17:40:51.281 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:50 smithi040 ceph-mon[27571]: osdmap e20: 3 total, 3 up, 3 in 2024-02-14T17:40:51.281 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:50 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:51.282 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:50 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:51.282 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:50 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:51.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:50 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-14T17:40:51.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:50 smithi096 ceph-mon[27796]: osdmap e20: 3 total, 3 up, 3 in 2024-02-14T17:40:51.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:50 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:51.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:50 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:51.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:50 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:51.362 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:40:50 smithi027 sudo[40520]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-02-14T17:40:51.362 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:40:50 smithi027 sudo[40520]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-14T17:40:51.363 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:40:50 smithi027 sudo[40520]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:40:51.363 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:40:50 smithi027 sudo[40520]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:40:51.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-14T17:40:51.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:50 smithi027 ceph-mon[25828]: osdmap e20: 3 total, 3 up, 3 in 2024-02-14T17:40:51.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:51.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:51.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:50 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:51.562 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:40:51 smithi040 sudo[34340]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme4n1 2024-02-14T17:40:51.563 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:40:51 smithi040 sudo[34340]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-14T17:40:51.563 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:40:51 smithi040 sudo[34340]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:40:51.563 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:40:51 smithi040 sudo[34340]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:40:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:40:51.795 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"932ae096-cb5f-11ee-95b9-87774f69a715","created":"2024-02-14T17:37:05.363563+0000","modified":"2024-02-14T17:40:50.994370+0000","last_up_change":"2024-02-14T17:40:47.973379+0000","last_in_change":"2024-02-14T17:40:28.275124+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-14T17:40:48.885325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"90eb8397-c65b-458e-a9fe-16e8b137a2ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1313296731},{"type":"v1","addr":"172.21.15.27:6803","nonce":1313296731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1313296731},{"type":"v1","addr":"172.21.15.27:6805","nonce":1313296731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1313296731},{"type":"v1","addr":"172.21.15.27:6809","nonce":1313296731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1313296731},{"type":"v1","addr":"172.21.15.27:6807","nonce":1313296731}]},"public_addr":"172.21.15.27:6803/1313296731","cluster_addr":"172.21.15.27:6805/1313296731","heartbeat_back_addr":"172.21.15.27:6809/1313296731","heartbeat_front_addr":"172.21.15.27:6807/1313296731","state":["exists","up"]},{"osd":1,"uuid":"a33d7701-d5f3-482e-8e5a-d658508b5468","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":4075500575},{"type":"v1","addr":"172.21.15.40:6801","nonce":4075500575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":4075500575},{"type":"v1","addr":"172.21.15.40:6803","nonce":4075500575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":4075500575},{"type":"v1","addr":"172.21.15.40:6807","nonce":4075500575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":4075500575},{"type":"v1","addr":"172.21.15.40:6805","nonce":4075500575}]},"public_addr":"172.21.15.40:6801/4075500575","cluster_addr":"172.21.15.40:6803/4075500575","heartbeat_back_addr":"172.21.15.40:6807/4075500575","heartbeat_front_addr":"172.21.15.40:6805/4075500575","state":["exists","up"]},{"osd":2,"uuid":"eac09c38-7d9b-43e1-8e02-7b99103304c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":1178359084},{"type":"v1","addr":"172.21.15.96:6801","nonce":1178359084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":1178359084},{"type":"v1","addr":"172.21.15.96:6803","nonce":1178359084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":1178359084},{"type":"v1","addr":"172.21.15.96:6807","nonce":1178359084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":1178359084},{"type":"v1","addr":"172.21.15.96:6805","nonce":1178359084}]},"public_addr":"172.21.15.96:6801/1178359084","cluster_addr":"172.21.15.96:6803/1178359084","heartbeat_back_addr":"172.21.15.96:6807/1178359084","heartbeat_front_addr":"172.21.15.96:6805/1178359084","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:39:55.426038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:40:20.620212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:40:45.953292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/681803954":"2024-02-15T17:38:12.353293+0000","172.21.15.27:0/3655809749":"2024-02-15T17:38:12.353293+0000","172.21.15.27:6801/2861113236":"2024-02-15T17:38:12.353293+0000","172.21.15.27:6800/2861113236":"2024-02-15T17:38:12.353293+0000","172.21.15.27:6801/4072918299":"2024-02-15T17:37:44.165328+0000","172.21.15.27:0/1727691481":"2024-02-15T17:37:44.165328+0000","172.21.15.27:6800/4072918299":"2024-02-15T17:37:44.165328+0000","172.21.15.27:0/3104063072":"2024-02-15T17:37:44.165328+0000","172.21.15.27:0/801052348":"2024-02-15T17:37:44.165328+0000","172.21.15.27:0/2483876909":"2024-02-15T17:38:12.353293+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-14T17:40:51.979 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:40:51 smithi040 sudo[34395]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-02-14T17:40:51.979 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:40:51 smithi040 sudo[34395]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-14T17:40:51.979 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:40:51 smithi040 sudo[34395]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:40:51.979 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:40:51 smithi040 sudo[34395]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:40:52.084 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:40:51 smithi096 sudo[35329]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme4n1 2024-02-14T17:40:52.084 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:40:51 smithi096 sudo[35329]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-14T17:40:52.085 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:40:51 smithi096 sudo[35329]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:40:52.085 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:40:51 smithi096 sudo[35329]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:40:52.085 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:52 smithi096 ceph-mon[27796]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:52.085 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:52 smithi096 ceph-mon[27796]: Detected new or changed devices on smithi096 2024-02-14T17:40:52.085 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:52 smithi096 ceph-mon[27796]: osdmap e21: 3 total, 3 up, 3 in 2024-02-14T17:40:52.085 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:52 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.085 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:52 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.085 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:52 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.085 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:52 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.085 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:52 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.086 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:52 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/4088060821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:40:52.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:52 smithi027 ceph-mon[25828]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:52.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:52 smithi027 ceph-mon[25828]: Detected new or changed devices on smithi096 2024-02-14T17:40:52.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:52 smithi027 ceph-mon[25828]: osdmap e21: 3 total, 3 up, 3 in 2024-02-14T17:40:52.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:52 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:52 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:52 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:52 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:52 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:52 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/4088060821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:40:52.417 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-14T17:40:48.885325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-14T17:40:52.418 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-14T17:40:52.472 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:40:52 smithi096 sudo[35340]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-02-14T17:40:52.472 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:40:52 smithi096 sudo[35340]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-14T17:40:52.472 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:40:52 smithi096 sudo[35340]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:40:52.472 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:40:52 smithi096 sudo[35340]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:40:52.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:52 smithi040 ceph-mon[27571]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:52.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:52 smithi040 ceph-mon[27571]: Detected new or changed devices on smithi096 2024-02-14T17:40:52.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:52 smithi040 ceph-mon[27571]: osdmap e21: 3 total, 3 up, 3 in 2024-02-14T17:40:52.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:52 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:52 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:52 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:52 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:52 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:52.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:52 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/4088060821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:40:52.671 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:40:52.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:52 smithi027 sudo[40629]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-14T17:40:52.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:52 smithi027 sudo[40629]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-14T17:40:52.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:52 smithi027 sudo[40629]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:40:52.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:52 smithi027 sudo[40629]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:40:53.106 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:53 smithi027 sudo[40708]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-14T17:40:53.106 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:53 smithi027 sudo[40708]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-14T17:40:53.107 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:53 smithi027 sudo[40708]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:40:53.107 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:53 smithi027 sudo[40708]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:40:53.107 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:53 smithi027 ceph-mon[25828]: mgrmap e16: y(active, since 2m), standbys: x, z 2024-02-14T17:40:53.107 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:53 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:53.107 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:53 smithi027 ceph-mon[25828]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:40:53.201 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:53 smithi096 ceph-mon[27796]: mgrmap e16: y(active, since 2m), standbys: x, z 2024-02-14T17:40:53.201 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:53 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:53.201 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:53 smithi096 ceph-mon[27796]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:40:53.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:53 smithi040 ceph-mon[27571]: mgrmap e16: y(active, since 2m), standbys: x, z 2024-02-14T17:40:53.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:53 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:53.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:53 smithi040 ceph-mon[27571]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:40:53.494 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:53 smithi096 sudo[35430]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-14T17:40:53.495 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:53 smithi096 sudo[35430]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-14T17:40:53.495 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:53 smithi096 sudo[35430]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:40:53.495 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:53 smithi096 sudo[35430]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:40:53.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:53 smithi096 sudo[35460]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-14T17:40:53.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:53 smithi096 sudo[35460]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-14T17:40:53.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:53 smithi096 sudo[35460]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:40:53.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:53 smithi096 sudo[35460]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:40:54.110 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:53 smithi040 sudo[34414]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-14T17:40:54.111 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:53 smithi040 sudo[34414]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-14T17:40:54.111 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:53 smithi040 sudo[34414]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:40:54.111 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:53 smithi040 sudo[34414]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:40:54.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: pgmap v91: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:54.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:40:54.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:40:54.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:40:54.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:40:54.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:54.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:40:54.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:40:54.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:40:54.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:40:54.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:40:54.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:54.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:40:54.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:40:54.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:54 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:40:54.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: pgmap v91: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:54.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:40:54.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:40:54.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:40:54.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:40:54.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:54.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:40:54.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:40:54.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:40:54.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:40:54.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:40:54.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:54.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:40:54.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:40:54.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:54 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:40:54.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 sudo[34417]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-14T17:40:54.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 sudo[34417]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-14T17:40:54.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 sudo[34417]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:40:54.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 sudo[34417]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:40:54.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: pgmap v91: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:54.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:40:54.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:40:54.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:40:54.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:40:54.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:54.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:40:54.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:40:54.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:40:54.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:40:54.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:40:54.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:54.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:40:54.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:40:54.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:54 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:40:54.617 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-02-14T17:40:55.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:55 smithi027 ceph-mon[25828]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:40:55.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:55 smithi027 ceph-mon[25828]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:40:55.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:55 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:55.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:55 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/2230357845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-14T17:40:55.357 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi027 2024-02-14T17:40:55.358 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi027=foo.a' 2024-02-14T17:40:55.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:55 smithi040 ceph-mon[27571]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:40:55.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:55 smithi040 ceph-mon[27571]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:40:55.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:55 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:55.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:55 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/2230357845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-14T17:40:55.391 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:55 smithi096 ceph-mon[27796]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:40:55.391 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:55 smithi096 ceph-mon[27796]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:40:55.391 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:55 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:55.391 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:55 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/2230357845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-14T17:40:55.608 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.c/config 2024-02-14T17:40:56.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:56 smithi096 ceph-mon[27796]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:56.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:56 smithi040 ceph-mon[27571]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:56.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:56 smithi027 ceph-mon[25828]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:57.432 INFO:teuthology.orchestra.run.smithi096.stdout:Scheduled rgw.foo.a update... 2024-02-14T17:40:58.079 DEBUG:teuthology.orchestra.run.smithi027:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@rgw.foo.a.service 2024-02-14T17:40:58.081 INFO:tasks.cephadm:Adding prometheus.a on smithi040 2024-02-14T17:40:58.081 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch apply prometheus '1;smithi040=a' 2024-02-14T17:40:58.352 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.c/config 2024-02-14T17:40:58.418 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:58 smithi027 ceph-mon[25828]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:58.418 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:58 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:58.418 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:58 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:58.418 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:58 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:58.418 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:58 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:58.418 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:58 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:58.418 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:58 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:40:58.418 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:58 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-14T17:40:58.418 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:58 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:58.419 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:58 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:58.474 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:58 smithi096 ceph-mon[27796]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:58.474 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:58 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:58.474 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:58 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:58.474 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:58 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:58.474 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:58 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:58.474 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:58 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:58.474 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:58 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:40:58.474 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:58 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-14T17:40:58.474 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:58 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:58.475 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:58 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:58.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:58 smithi040 ceph-mon[27571]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:40:58.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:58 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:58.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:58 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:58.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:58 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:58.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:58 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:58.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:58 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:58.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:58 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:40:58.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:58 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-14T17:40:58.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:58 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:40:58.481 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:58 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:59.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:59 smithi040 ceph-mon[27571]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi027=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:59.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:59 smithi040 ceph-mon[27571]: Saving service rgw.foo.a spec with placement smithi027=foo.a;count:1 2024-02-14T17:40:59.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:40:59 smithi040 ceph-mon[27571]: Deploying daemon rgw.foo.a on smithi027 2024-02-14T17:40:59.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:59 smithi027 ceph-mon[25828]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi027=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:59.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:59 smithi027 ceph-mon[25828]: Saving service rgw.foo.a spec with placement smithi027=foo.a;count:1 2024-02-14T17:40:59.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:40:59 smithi027 ceph-mon[25828]: Deploying daemon rgw.foo.a on smithi027 2024-02-14T17:40:59.484 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:59 smithi096 ceph-mon[27796]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi027=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:59.484 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:59 smithi096 ceph-mon[27796]: Saving service rgw.foo.a spec with placement smithi027=foo.a;count:1 2024-02-14T17:40:59.484 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:40:59 smithi096 ceph-mon[27796]: Deploying daemon rgw.foo.a on smithi027 2024-02-14T17:40:59.891 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:40:59 smithi027 systemd[1]: Starting Ceph rgw.foo.a for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:41:00.017 INFO:teuthology.orchestra.run.smithi096.stdout:Scheduled prometheus update... 2024-02-14T17:41:00.151 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:40:59 smithi027 podman[41146]: 2024-02-14 17:40:59.792511008 +0000 UTC m=+0.018043275 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:41:00.152 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:40:59 smithi027 podman[41146]: 2024-02-14 17:40:59.914410285 +0000 UTC m=+0.139942542 container create 77b6b21cb6a8e1052f332558a9d47a0618085ee47004e70c2e526ccaa0e136d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5) 2024-02-14T17:41:00.152 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:41:00 smithi027 podman[41146]: 2024-02-14 17:41:00.132964581 +0000 UTC m=+0.358496836 container init 77b6b21cb6a8e1052f332558a9d47a0618085ee47004e70c2e526ccaa0e136d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-14T17:41:00.152 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:41:00 smithi027 podman[41146]: 2024-02-14 17:41:00.138932102 +0000 UTC m=+0.364464357 container start 77b6b21cb6a8e1052f332558a9d47a0618085ee47004e70c2e526ccaa0e136d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-rgw-foo-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-14T17:41:00.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:00 smithi096 ceph-mon[27796]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:00.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:00 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:00.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:00 smithi040 ceph-mon[27571]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:00.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:00 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:00.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:00 smithi027 ceph-mon[25828]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:00.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:00 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:00.483 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:41:00 smithi027 bash[41146]: 77b6b21cb6a8e1052f332558a9d47a0618085ee47004e70c2e526ccaa0e136d4 2024-02-14T17:41:00.483 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:41:00 smithi027 systemd[1]: Started Ceph rgw.foo.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:41:00.643 DEBUG:teuthology.orchestra.run.smithi040:prometheus.a> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@prometheus.a.service 2024-02-14T17:41:00.646 INFO:tasks.cephadm:Adding node-exporter.a on smithi027 2024-02-14T17:41:00.646 INFO:tasks.cephadm:Adding node-exporter.b on smithi040 2024-02-14T17:41:00.646 INFO:tasks.cephadm:Adding node-exporter.c on smithi096 2024-02-14T17:41:00.647 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch apply node-exporter '3;smithi027=a;smithi040=b;smithi096=c' 2024-02-14T17:41:00.878 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.c/config 2024-02-14T17:41:01.664 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:01 smithi096 ceph-mon[27796]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi040=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:01.664 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:01 smithi096 ceph-mon[27796]: Saving service prometheus spec with placement smithi040=a;count:1 2024-02-14T17:41:01.664 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.664 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.665 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.665 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.665 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.665 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:01.665 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:01.665 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:01 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:01.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:01 smithi027 ceph-mon[25828]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi040=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:01.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:01 smithi027 ceph-mon[25828]: Saving service prometheus spec with placement smithi040=a;count:1 2024-02-14T17:41:01.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:01 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:01 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:01 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:01 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:01 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:01 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:01.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:01 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:01.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:01 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:01.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:01 smithi040 ceph-mon[27571]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi040=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:01.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:01 smithi040 ceph-mon[27571]: Saving service prometheus spec with placement smithi040=a;count:1 2024-02-14T17:41:01.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:01 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:01 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:01 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:01 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:01 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:01.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:01 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:01.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:01 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:01.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:01 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:02.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:02 smithi096 ceph-mon[27796]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-02-14T17:41:02.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:02 smithi096 ceph-mon[27796]: Saving service rgw.foo.a spec with placement smithi027=foo.a;count:1 2024-02-14T17:41:02.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:02 smithi096 ceph-mon[27796]: Checking dashboard <-> RGW credentials 2024-02-14T17:41:02.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:02 smithi096 ceph-mon[27796]: osdmap e22: 3 total, 3 up, 3 in 2024-02-14T17:41:02.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:02 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/1671935243' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-14T17:41:02.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:02 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/542579166' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-14T17:41:02.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:02 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:02.666 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:02 smithi027 ceph-mon[25828]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-02-14T17:41:02.666 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:02 smithi027 ceph-mon[25828]: Saving service rgw.foo.a spec with placement smithi027=foo.a;count:1 2024-02-14T17:41:02.666 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:02 smithi027 ceph-mon[25828]: Checking dashboard <-> RGW credentials 2024-02-14T17:41:02.666 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:02 smithi027 ceph-mon[25828]: osdmap e22: 3 total, 3 up, 3 in 2024-02-14T17:41:02.667 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:02 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/1671935243' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-14T17:41:02.667 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:02 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/542579166' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-14T17:41:02.667 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:02 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:02.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:02 smithi040 ceph-mon[27571]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-02-14T17:41:02.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:02 smithi040 ceph-mon[27571]: Saving service rgw.foo.a spec with placement smithi027=foo.a;count:1 2024-02-14T17:41:02.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:02 smithi040 ceph-mon[27571]: Checking dashboard <-> RGW credentials 2024-02-14T17:41:02.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:02 smithi040 ceph-mon[27571]: osdmap e22: 3 total, 3 up, 3 in 2024-02-14T17:41:02.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:02 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/1671935243' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-14T17:41:02.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:02 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/542579166' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-14T17:41:02.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:02 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:02.890 INFO:teuthology.orchestra.run.smithi096.stdout:Scheduled node-exporter update... 2024-02-14T17:41:03.479 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:03 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/1671935243' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-14T17:41:03.479 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:03 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/542579166' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-14T17:41:03.479 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:03 smithi096 ceph-mon[27796]: osdmap e23: 3 total, 3 up, 3 in 2024-02-14T17:41:03.479 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:03 smithi096 ceph-mon[27796]: pgmap v98: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-02-14T17:41:03.479 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:03 smithi096 ceph-mon[27796]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi027=a;smithi040=b;smithi096=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:03.479 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:03 smithi096 ceph-mon[27796]: Saving service node-exporter spec with placement smithi027=a;smithi040=b;smithi096=c;count:3 2024-02-14T17:41:03.479 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:03 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:03.479 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:03 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:03.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:03 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/1671935243' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-14T17:41:03.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:03 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/542579166' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-14T17:41:03.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:03 smithi027 ceph-mon[25828]: osdmap e23: 3 total, 3 up, 3 in 2024-02-14T17:41:03.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:03 smithi027 ceph-mon[25828]: pgmap v98: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-02-14T17:41:03.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:03 smithi027 ceph-mon[25828]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi027=a;smithi040=b;smithi096=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:03.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:03 smithi027 ceph-mon[25828]: Saving service node-exporter spec with placement smithi027=a;smithi040=b;smithi096=c;count:3 2024-02-14T17:41:03.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:03 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:03.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:03 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:03.504 DEBUG:teuthology.orchestra.run.smithi027:node-exporter.a> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@node-exporter.a.service 2024-02-14T17:41:03.507 DEBUG:teuthology.orchestra.run.smithi040:node-exporter.b> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@node-exporter.b.service 2024-02-14T17:41:03.510 DEBUG:teuthology.orchestra.run.smithi096:node-exporter.c> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@node-exporter.c.service 2024-02-14T17:41:03.513 INFO:tasks.cephadm:Adding alertmanager.a on smithi027 2024-02-14T17:41:03.513 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch apply alertmanager '1;smithi027=a' 2024-02-14T17:41:03.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:03 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/1671935243' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-14T17:41:03.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:03 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/542579166' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-14T17:41:03.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:03 smithi040 ceph-mon[27571]: osdmap e23: 3 total, 3 up, 3 in 2024-02-14T17:41:03.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:03 smithi040 ceph-mon[27571]: pgmap v98: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-02-14T17:41:03.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:03 smithi040 ceph-mon[27571]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi027=a;smithi040=b;smithi096=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:03.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:03 smithi040 ceph-mon[27571]: Saving service node-exporter spec with placement smithi027=a;smithi040=b;smithi096=c;count:3 2024-02-14T17:41:03.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:03 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:03.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:03 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:03.782 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.c/config 2024-02-14T17:41:04.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:04 smithi027 ceph-mon[25828]: osdmap e24: 3 total, 3 up, 3 in 2024-02-14T17:41:04.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:04 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:41:04.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:04 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/80821491' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:41:04.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:04 smithi027 ceph-mon[25828]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:41:04.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:04 smithi096 ceph-mon[27796]: osdmap e24: 3 total, 3 up, 3 in 2024-02-14T17:41:04.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:04 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:41:04.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:04 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/80821491' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:41:04.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:04 smithi096 ceph-mon[27796]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:41:04.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:04 smithi040 ceph-mon[27571]: osdmap e24: 3 total, 3 up, 3 in 2024-02-14T17:41:04.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:04 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:41:04.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:04 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/80821491' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:41:04.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:04 smithi040 ceph-mon[27571]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:41:05.538 INFO:teuthology.orchestra.run.smithi096.stdout:Scheduled alertmanager update... 2024-02-14T17:41:05.538 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:05 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-14T17:41:05.538 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:05 smithi096 ceph-mon[27796]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-14T17:41:05.539 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:05 smithi096 ceph-mon[27796]: osdmap e25: 3 total, 3 up, 3 in 2024-02-14T17:41:05.539 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:05 smithi096 ceph-mon[27796]: pgmap v101: 65 pgs: 11 creating+peering, 33 active+clean, 21 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-14T17:41:05.539 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:05 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:05.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:05 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-14T17:41:05.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:05 smithi040 ceph-mon[27571]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-14T17:41:05.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:05 smithi040 ceph-mon[27571]: osdmap e25: 3 total, 3 up, 3 in 2024-02-14T17:41:05.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:05 smithi040 ceph-mon[27571]: pgmap v101: 65 pgs: 11 creating+peering, 33 active+clean, 21 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-14T17:41:05.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:05 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:05.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:05 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-14T17:41:05.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:05 smithi027 ceph-mon[25828]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-14T17:41:05.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:05 smithi027 ceph-mon[25828]: osdmap e25: 3 total, 3 up, 3 in 2024-02-14T17:41:05.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:05 smithi027 ceph-mon[25828]: pgmap v101: 65 pgs: 11 creating+peering, 33 active+clean, 21 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-14T17:41:05.734 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:05 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:06.116 DEBUG:teuthology.orchestra.run.smithi027:alertmanager.a> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@alertmanager.a.service 2024-02-14T17:41:06.119 INFO:tasks.cephadm:Adding grafana.a on smithi040 2024-02-14T17:41:06.119 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph orch apply grafana '1;smithi040=a' 2024-02-14T17:41:06.349 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.c/config 2024-02-14T17:41:06.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:06 smithi027 ceph-mon[25828]: osdmap e26: 3 total, 3 up, 3 in 2024-02-14T17:41:06.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:06 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:41:06.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:06 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/80821491' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:41:06.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:06 smithi027 ceph-mon[25828]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:41:06.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:06 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:06.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:06 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:06.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:06 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:06.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:06 smithi027 ceph-mon[25828]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi027=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:06.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:06 smithi027 ceph-mon[25828]: Saving service alertmanager spec with placement smithi027=a;count:1 2024-02-14T17:41:06.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:06 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:06.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:06 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-14T17:41:06.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:06 smithi027 ceph-mon[25828]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-14T17:41:06.726 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:06 smithi096 ceph-mon[27796]: osdmap e26: 3 total, 3 up, 3 in 2024-02-14T17:41:06.727 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:06 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:41:06.727 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:06 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/80821491' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:41:06.727 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:06 smithi096 ceph-mon[27796]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:41:06.727 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:06 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:06.727 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:06 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:06.727 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:06 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:06.727 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:06 smithi096 ceph-mon[27796]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi027=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:06.727 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:06 smithi096 ceph-mon[27796]: Saving service alertmanager spec with placement smithi027=a;count:1 2024-02-14T17:41:06.727 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:06 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:06.728 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:06 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-14T17:41:06.728 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:06 smithi096 ceph-mon[27796]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-14T17:41:06.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:06 smithi040 ceph-mon[27571]: osdmap e26: 3 total, 3 up, 3 in 2024-02-14T17:41:06.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:06 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:41:06.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:06 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/80821491' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:41:06.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:06 smithi040 ceph-mon[27571]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:41:06.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:06 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:06.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:06 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:06.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:06 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:06.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:06 smithi040 ceph-mon[27571]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi027=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:06.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:06 smithi040 ceph-mon[27571]: Saving service alertmanager spec with placement smithi027=a;count:1 2024-02-14T17:41:06.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:06 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:06.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:06 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-14T17:41:06.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:06 smithi040 ceph-mon[27571]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-14T17:41:07.448 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:07 smithi096 ceph-mon[27796]: pgmap v103: 97 pgs: 11 creating+peering, 33 active+clean, 53 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-14T17:41:07.448 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:07 smithi096 ceph-mon[27796]: osdmap e27: 3 total, 3 up, 3 in 2024-02-14T17:41:07.448 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:07 smithi096 ceph-mon[27796]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-14T17:41:07.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:07 smithi027 ceph-mon[25828]: pgmap v103: 97 pgs: 11 creating+peering, 33 active+clean, 53 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-14T17:41:07.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:07 smithi027 ceph-mon[25828]: osdmap e27: 3 total, 3 up, 3 in 2024-02-14T17:41:07.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:07 smithi027 ceph-mon[25828]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-14T17:41:07.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:07 smithi040 ceph-mon[27571]: pgmap v103: 97 pgs: 11 creating+peering, 33 active+clean, 53 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-14T17:41:07.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:07 smithi040 ceph-mon[27571]: osdmap e27: 3 total, 3 up, 3 in 2024-02-14T17:41:07.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:07 smithi040 ceph-mon[27571]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-14T17:41:08.131 INFO:teuthology.orchestra.run.smithi096.stdout:Scheduled grafana update... 2024-02-14T17:41:08.686 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:08 smithi096 ceph-mon[27796]: osdmap e28: 3 total, 3 up, 3 in 2024-02-14T17:41:08.686 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:08 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-14T17:41:08.686 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:08 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/80821491' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-14T17:41:08.686 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:08 smithi096 ceph-mon[27796]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-14T17:41:08.686 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:08 smithi096 ceph-mon[27796]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi040=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:08.686 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:08 smithi096 ceph-mon[27796]: Saving service grafana spec with placement smithi040=a;count:1 2024-02-14T17:41:08.686 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:08 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:08.686 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:08 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-14T17:41:08.711 DEBUG:teuthology.orchestra.run.smithi040:grafana.a> sudo journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@grafana.a.service 2024-02-14T17:41:08.714 INFO:tasks.cephadm:Setting up client nodes... 2024-02-14T17:41:08.714 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-14T17:41:08.728 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:08 smithi040 ceph-mon[27571]: osdmap e28: 3 total, 3 up, 3 in 2024-02-14T17:41:08.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:08 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-14T17:41:08.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:08 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/80821491' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-14T17:41:08.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:08 smithi040 ceph-mon[27571]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-14T17:41:08.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:08 smithi040 ceph-mon[27571]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi040=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:08.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:08 smithi040 ceph-mon[27571]: Saving service grafana spec with placement smithi040=a;count:1 2024-02-14T17:41:08.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:08 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:08.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:08 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-14T17:41:08.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:08 smithi027 ceph-mon[25828]: osdmap e28: 3 total, 3 up, 3 in 2024-02-14T17:41:08.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:08 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-14T17:41:08.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:08 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/80821491' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-14T17:41:08.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:08 smithi027 ceph-mon[25828]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-14T17:41:08.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:08 smithi027 ceph-mon[25828]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi040=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:08.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:08 smithi027 ceph-mon[25828]: Saving service grafana spec with placement smithi040=a;count:1 2024-02-14T17:41:08.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:08 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:08.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:08 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-14T17:41:08.974 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:41:09.728 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:09 smithi040 ceph-mon[27571]: pgmap v106: 129 pgs: 30 creating+peering, 77 active+clean, 22 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-14T17:41:09.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:09 smithi040 ceph-mon[27571]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-14T17:41:09.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:09 smithi040 ceph-mon[27571]: osdmap e29: 3 total, 3 up, 3 in 2024-02-14T17:41:09.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:09 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-14T17:41:09.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:09 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/80821491' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-14T17:41:09.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:09 smithi040 ceph-mon[27571]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-14T17:41:09.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:09 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-14T17:41:09.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:09 smithi027 ceph-mon[25828]: pgmap v106: 129 pgs: 30 creating+peering, 77 active+clean, 22 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-14T17:41:09.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:09 smithi027 ceph-mon[25828]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-14T17:41:09.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:09 smithi027 ceph-mon[25828]: osdmap e29: 3 total, 3 up, 3 in 2024-02-14T17:41:09.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:09 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-14T17:41:09.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:09 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/80821491' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-14T17:41:09.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:09 smithi027 ceph-mon[25828]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-14T17:41:09.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:09 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-14T17:41:09.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:09 smithi096 ceph-mon[27796]: pgmap v106: 129 pgs: 30 creating+peering, 77 active+clean, 22 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-14T17:41:09.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:09 smithi096 ceph-mon[27796]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-14T17:41:09.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:09 smithi096 ceph-mon[27796]: osdmap e29: 3 total, 3 up, 3 in 2024-02-14T17:41:09.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:09 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-14T17:41:09.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:09 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/80821491' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-14T17:41:09.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:09 smithi096 ceph-mon[27796]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-14T17:41:09.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:09 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/694868977' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-14T17:41:10.462 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:10 smithi027 ceph-mon[25828]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-14T17:41:10.462 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:10 smithi027 ceph-mon[25828]: osdmap e30: 3 total, 3 up, 3 in 2024-02-14T17:41:10.462 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:10 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:10.462 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:10 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:10.462 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:10 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:10.462 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:10 smithi027 ceph-mon[25828]: Deploying daemon prometheus.a on smithi040 2024-02-14T17:41:10.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:10 smithi040 ceph-mon[27571]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-14T17:41:10.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:10 smithi040 ceph-mon[27571]: osdmap e30: 3 total, 3 up, 3 in 2024-02-14T17:41:10.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:10 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:10.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:10 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:10.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:10 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:10.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:10 smithi040 ceph-mon[27571]: Deploying daemon prometheus.a on smithi040 2024-02-14T17:41:10.788 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:10 smithi096 ceph-mon[27796]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-14T17:41:10.788 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:10 smithi096 ceph-mon[27796]: osdmap e30: 3 total, 3 up, 3 in 2024-02-14T17:41:10.788 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:10 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:10.788 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:10 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:10.788 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:10 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:10.789 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:10 smithi096 ceph-mon[27796]: Deploying daemon prometheus.a on smithi040 2024-02-14T17:41:10.989 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-02-14T17:41:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQA2+8xl6mjFOhAAhQ6PDjjiTX4Z0XNH4SDKjw== 2024-02-14T17:41:11.622 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:11 smithi027 ceph-mon[25828]: pgmap v109: 129 pgs: 19 creating+peering, 110 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-14T17:41:11.623 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:11 smithi027 ceph-mon[25828]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-14T17:41:11.623 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:11 smithi027 ceph-mon[25828]: Cluster is now healthy 2024-02-14T17:41:11.623 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:11 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:11.623 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:11 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:11.623 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:11 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:11.623 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:11 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/857379157' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:11.623 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:11 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/857379157' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:41:11.655 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:41:11.655 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-14T17:41:11.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-14T17:41:11.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:11 smithi040 ceph-mon[27571]: pgmap v109: 129 pgs: 19 creating+peering, 110 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-14T17:41:11.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:11 smithi040 ceph-mon[27571]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-14T17:41:11.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:11 smithi040 ceph-mon[27571]: Cluster is now healthy 2024-02-14T17:41:11.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:11.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:11.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:11 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:11.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:11 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/857379157' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:11.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:11 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/857379157' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:41:11.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-14T17:41:11.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:11 smithi096 ceph-mon[27796]: pgmap v109: 129 pgs: 19 creating+peering, 110 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-14T17:41:11.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:11 smithi096 ceph-mon[27796]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-14T17:41:11.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:11 smithi096 ceph-mon[27796]: Cluster is now healthy 2024-02-14T17:41:11.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:11 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:11.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:11 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:11.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:11 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:11.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:11 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/857379157' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:11.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:11 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/857379157' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:41:12.018 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.b/config 2024-02-14T17:41:12.947 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:12 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:12.947 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:12 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:12.947 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:12 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:12.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:12 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:12.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:12 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:12.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:12 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:13.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:13.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:13.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:12 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:13.942 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:13 smithi040 ceph-mon[27571]: pgmap v110: 129 pgs: 19 creating+peering, 110 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2 KiB/s rd, 341 B/s wr, 3 op/s 2024-02-14T17:41:13.985 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:13 smithi096 ceph-mon[27796]: pgmap v110: 129 pgs: 19 creating+peering, 110 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2 KiB/s rd, 341 B/s wr, 3 op/s 2024-02-14T17:41:14.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:13 smithi027 ceph-mon[25828]: pgmap v110: 129 pgs: 19 creating+peering, 110 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2 KiB/s rd, 341 B/s wr, 3 op/s 2024-02-14T17:41:14.672 INFO:teuthology.orchestra.run.smithi040.stdout:[client.1] 2024-02-14T17:41:14.672 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQA6+8xljce/JxAAgCa6sTLSgCugUu1fk7Qiuw== 2024-02-14T17:41:14.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:14 smithi040 ceph-mon[27571]: from='client.? 172.21.15.40:0/2852657409' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:14.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:14 smithi040 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:14.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:14 smithi040 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:41:15.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:14 smithi027 ceph-mon[25828]: from='client.? 172.21.15.40:0/2852657409' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:15.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:14 smithi027 ceph-mon[25828]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:15.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:14 smithi027 ceph-mon[25828]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:41:15.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:14 smithi096 ceph-mon[27796]: from='client.? 172.21.15.40:0/2852657409' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:15.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:14 smithi096 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:15.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:14 smithi096 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:41:16.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:15 smithi040 ceph-mon[27571]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 67 KiB/s rd, 5.1 KiB/s wr, 162 op/s 2024-02-14T17:41:16.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:15 smithi027 ceph-mon[25828]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 67 KiB/s rd, 5.1 KiB/s wr, 162 op/s 2024-02-14T17:41:16.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:15 smithi096 ceph-mon[27796]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 67 KiB/s rd, 5.1 KiB/s wr, 162 op/s 2024-02-14T17:41:16.544 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-02-14T17:41:16.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-14T17:41:16.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-14T17:41:16.588 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-14T17:41:16.836 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.c/config 2024-02-14T17:41:18.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:17 smithi096 ceph-mon[27796]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-02-14T17:41:18.104 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:17 smithi040 ceph-mon[27571]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-02-14T17:41:18.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:17 smithi027 ceph-mon[25828]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-02-14T17:41:18.707 INFO:teuthology.orchestra.run.smithi096.stdout:[client.2] 2024-02-14T17:41:18.707 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQA++8xl4DLQKRAAeUiH4yde50mqs7D1DjM72w== 2024-02-14T17:41:19.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:18 smithi096 ceph-mon[27796]: from='client.? 172.21.15.96:0/4161302402' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:19.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:18 smithi096 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:19.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:18 smithi096 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:41:19.153 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:18 smithi040 ceph-mon[27571]: from='client.? 172.21.15.96:0/4161302402' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:19.153 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:18 smithi040 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:19.153 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:18 smithi040 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:41:19.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:18 smithi027 ceph-mon[25828]: from='client.? 172.21.15.96:0/4161302402' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:19.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:18 smithi027 ceph-mon[25828]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:41:19.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:18 smithi027 ceph-mon[25828]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:41:19.493 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-02-14T17:41:19.493 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-02-14T17:41:19.493 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-02-14T17:41:19.537 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-14T17:41:19.538 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-14T17:41:19.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph mgr dump --format=json 2024-02-14T17:41:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:41:20.048 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:19 smithi040 ceph-mon[27571]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-02-14T17:41:20.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:19 smithi027 ceph-mon[25828]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-02-14T17:41:20.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:19 smithi096 ceph-mon[27796]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-02-14T17:41:21.809 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:41:22.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:21 smithi027 ceph-mon[25828]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 42 KiB/s rd, 3.1 KiB/s wr, 101 op/s 2024-02-14T17:41:22.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:21 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/2268420762' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-14T17:41:22.205 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:21 smithi040 ceph-mon[27571]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 42 KiB/s rd, 3.1 KiB/s wr, 101 op/s 2024-02-14T17:41:22.205 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:21 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/2268420762' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-14T17:41:22.206 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 podman[35537]: 2024-02-14 17:41:22.090870732 +0000 UTC m=+0.114863206 container create 1f71374f1913cfb359b000f830d6fa43662db0b7577d95d221633f8e1afa3e4f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:22.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:21 smithi096 ceph-mon[27796]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 42 KiB/s rd, 3.1 KiB/s wr, 101 op/s 2024-02-14T17:41:22.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:21 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/2268420762' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-14T17:41:22.595 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 podman[35537]: 2024-02-14 17:41:22.309697059 +0000 UTC m=+0.333689525 container init 1f71374f1913cfb359b000f830d6fa43662db0b7577d95d221633f8e1afa3e4f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:22.595 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 podman[35537]: 2024-02-14 17:41:22.315548958 +0000 UTC m=+0.339541433 container start 1f71374f1913cfb359b000f830d6fa43662db0b7577d95d221633f8e1afa3e4f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:22.595 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.339Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-02-14T17:41:22.595 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.339Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-02-14T17:41:22.595 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.339Z caller=main.go:570 level=info host_details="(Linux 5.14.0-420.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 9 18:10:28 UTC 2024 x86_64 smithi040 (none))" 2024-02-14T17:41:22.595 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.339Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.339Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.341Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.342Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.343Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.343Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.344Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.344Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.471µs 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.344Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.344Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.344Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.549µs wal_replay_duration=202.738µs wbl_replay_duration=104ns total_replay_duration=236.975µs 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.345Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.345Z caller=main.go:1029 level=info msg="TSDB started" 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.345Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-14T17:41:22.596 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.364Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.746678ms db_storage=859ns remote_storage=1.046µs web_handler=268ns query_engine=775ns scrape=444.7µs scrape_sd=57.82µs notify=1.068µs notify_sd=2.345µs rules=17.88315ms tracing=5.485µs 2024-02-14T17:41:22.597 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.364Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-02-14T17:41:22.597 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:41:22.364Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-02-14T17:41:22.597 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 bash[35537]: 1f71374f1913cfb359b000f830d6fa43662db0b7577d95d221633f8e1afa3e4f 2024-02-14T17:41:22.597 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:41:22 smithi040 systemd[1]: Started Ceph prometheus.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:41:22.706 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1373248867},{"type":"v1","addr":"172.21.15.27:6801","nonce":1373248867}]},"active_addr":"172.21.15.27:6801/1373248867","active_change":"2024-02-14T17:38:12.353455+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24107,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.27:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":4003650419}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2642557763}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2849862895}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3305096772}]}]} 2024-02-14T17:41:22.711 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-14T17:41:22.712 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-14T17:41:22.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-14T17:41:23.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:23 smithi027 ceph-mon[25828]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.8 KiB/s wr, 92 op/s 2024-02-14T17:41:23.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:23 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:23.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:23 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:23.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:23 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:23.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:23 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-14T17:41:23.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:23 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:23.592 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:23 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: ignoring --setuser ceph since I am not root 2024-02-14T17:41:23.592 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:23 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: ignoring --setgroup ceph since I am not root 2024-02-14T17:41:23.729 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: ignoring --setuser ceph since I am not root 2024-02-14T17:41:23.729 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: ignoring --setgroup ceph since I am not root 2024-02-14T17:41:23.729 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:23 smithi040 ceph-mgr[29265]: -- 172.21.15.40:0/3510522977 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55aabf5d21a0 con 0x55aabf5a9000 2024-02-14T17:41:23.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:23 smithi040 ceph-mon[27571]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.8 KiB/s wr, 92 op/s 2024-02-14T17:41:23.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:23 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:23.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:23 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:23.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:23 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:23.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:23 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-14T17:41:23.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:23 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:23.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:23 smithi096 ceph-mon[27796]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.8 KiB/s wr, 92 op/s 2024-02-14T17:41:23.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:23 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:23.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:23 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:23.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:23 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:23.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:23 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-14T17:41:23.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:23 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' 2024-02-14T17:41:23.796 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:23 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: ignoring --setuser ceph since I am not root 2024-02-14T17:41:23.796 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:23 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: ignoring --setgroup ceph since I am not root 2024-02-14T17:41:23.796 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:23 smithi096 ceph-mgr[30163]: -- 172.21.15.96:0/1132190805 <== mon.1 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e5bb0161a0 con 0x55e5bafed000 2024-02-14T17:41:24.140 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:41:24.191 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:24 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:24.086+0000 7ff0d3c1a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:41:24.191 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:24 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:24.190+0000 7ff0d3c1a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:41:24.295 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:24 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:24.039+0000 7f99305ad200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:41:24.295 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:24 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:24.133+0000 7f99305ad200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:41:24.470 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:24 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:24.048+0000 7f0094b73200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:41:24.470 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:24 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:24.144+0000 7f0094b73200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:41:24.728 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:24 smithi027 ceph-mon[25828]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-14T17:41:24.728 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:24 smithi027 ceph-mon[25828]: mgrmap e17: y(active, since 3m), standbys: x, z 2024-02-14T17:41:24.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:24 smithi040 ceph-mon[27571]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-14T17:41:24.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:24 smithi040 ceph-mon[27571]: mgrmap e17: y(active, since 3m), standbys: x, z 2024-02-14T17:41:24.766 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:24 smithi096 ceph-mon[27796]: from='mgr.14152 172.21.15.27:0/1575100980' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-14T17:41:24.766 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:24 smithi096 ceph-mon[27796]: mgrmap e17: y(active, since 3m), standbys: x, z 2024-02-14T17:41:24.983 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:24 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:24.918+0000 7ff0d3c1a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:41:25.045 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:24 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:24.764+0000 7f99305ad200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:41:25.045 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:24 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:24.855+0000 7f99305ad200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:41:25.045 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:24 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:24.992+0000 7f99305ad200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:41:25.140 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:24 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:24.801+0000 7f0094b73200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:41:25.140 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:24 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:24.896+0000 7f0094b73200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:41:25.140 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:25 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:25.039+0000 7f0094b73200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:41:25.140 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:25 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:25.138+0000 7f0094b73200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:41:25.273 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:25.015+0000 7ff0d3c1a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:41:25.273 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:25.171+0000 7ff0d3c1a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:41:25.274 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:25.272+0000 7ff0d3c1a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:41:25.424 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:25 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:25.086+0000 7f99305ad200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:41:25.425 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:25 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:25.167+0000 7f99305ad200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:41:25.425 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:25 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:25.423+0000 7f99305ad200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:41:25.478 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:25 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:25.224+0000 7f0094b73200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:41:25.530 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:25.360+0000 7ff0d3c1a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:41:25.712 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:25 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:25.532+0000 7f99305ad200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:41:25.712 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:25 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:25.616+0000 7f99305ad200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:41:25.712 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:25 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:25.710+0000 7f99305ad200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:41:25.797 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:25 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:25.497+0000 7f0094b73200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:41:25.797 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:25 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:25.610+0000 7f0094b73200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:41:25.797 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:25 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:25.698+0000 7f0094b73200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:41:25.797 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:25 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:25.795+0000 7f0094b73200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:41:25.805 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:25.653+0000 7ff0d3c1a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:41:25.805 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:25.773+0000 7ff0d3c1a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:41:26.045 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:25 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:25.884+0000 7f99305ad200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:41:26.151 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:25.864+0000 7ff0d3c1a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:41:26.151 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:25.964+0000 7ff0d3c1a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:41:26.151 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:26 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:26.149+0000 7ff0d3c1a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:41:26.157 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:25 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:25.973+0000 7f0094b73200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:41:26.871 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:41:26.872 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"fsid":"932ae096-cb5f-11ee-95b9-87774f69a715","created":"2024-02-14T17:37:05.363563+0000","modified":"2024-02-14T17:41:09.396347+0000","last_up_change":"2024-02-14T17:40:47.973379+0000","last_in_change":"2024-02-14T17:40:28.275124+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-14T17:40:48.885325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-14T17:41:00.605527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-02-14T17:41:02.879326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-02-14T17:41:04.830834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-02-14T17:41:06.900998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"90eb8397-c65b-458e-a9fe-16e8b137a2ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1313296731},{"type":"v1","addr":"172.21.15.27:6803","nonce":1313296731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1313296731},{"type":"v1","addr":"172.21.15.27:6805","nonce":1313296731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1313296731},{"type":"v1","addr":"172.21.15.27:6809","nonce":1313296731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1313296731},{"type":"v1","addr":"172.21.15.27:6807","nonce":1313296731}]},"public_addr":"172.21.15.27:6803/1313296731","cluster_addr":"172.21.15.27:6805/1313296731","heartbeat_back_addr":"172.21.15.27:6809/1313296731","heartbeat_front_addr":"172.21.15.27:6807/1313296731","state":["exists","up"]},{"osd":1,"uuid":"a33d7701-d5f3-482e-8e5a-d658508b5468","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":4075500575},{"type":"v1","addr":"172.21.15.40:6801","nonce":4075500575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":4075500575},{"type":"v1","addr":"172.21.15.40:6803","nonce":4075500575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":4075500575},{"type":"v1","addr":"172.21.15.40:6807","nonce":4075500575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":4075500575},{"type":"v1","addr":"172.21.15.40:6805","nonce":4075500575}]},"public_addr":"172.21.15.40:6801/4075500575","cluster_addr":"172.21.15.40:6803/4075500575","heartbeat_back_addr":"172.21.15.40:6807/4075500575","heartbeat_front_addr":"172.21.15.40:6805/4075500575","state":["exists","up"]},{"osd":2,"uuid":"eac09c38-7d9b-43e1-8e02-7b99103304c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":1178359084},{"type":"v1","addr":"172.21.15.96:6801","nonce":1178359084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":1178359084},{"type":"v1","addr":"172.21.15.96:6803","nonce":1178359084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":1178359084},{"type":"v1","addr":"172.21.15.96:6807","nonce":1178359084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":1178359084},{"type":"v1","addr":"172.21.15.96:6805","nonce":1178359084}]},"public_addr":"172.21.15.96:6801/1178359084","cluster_addr":"172.21.15.96:6803/1178359084","heartbeat_back_addr":"172.21.15.96:6807/1178359084","heartbeat_front_addr":"172.21.15.96:6805/1178359084","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:39:55.426038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:40:20.620212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:40:45.953292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/681803954":"2024-02-15T17:38:12.353293+0000","172.21.15.27:0/3655809749":"2024-02-15T17:38:12.353293+0000","172.21.15.27:6801/2861113236":"2024-02-15T17:38:12.353293+0000","172.21.15.27:6800/2861113236":"2024-02-15T17:38:12.353293+0000","172.21.15.27:6801/4072918299":"2024-02-15T17:37:44.165328+0000","172.21.15.27:0/1727691481":"2024-02-15T17:37:44.165328+0000","172.21.15.27:6800/4072918299":"2024-02-15T17:37:44.165328+0000","172.21.15.27:0/3104063072":"2024-02-15T17:37:44.165328+0000","172.21.15.27:0/801052348":"2024-02-15T17:37:44.165328+0000","172.21.15.27:0/2483876909":"2024-02-15T17:38:12.353293+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-14T17:41:26.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:26 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/3687424477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:41:27.228 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:26 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/3687424477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:41:27.229 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:26 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:26.985+0000 7f0094b73200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:41:27.229 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:27 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:27.121+0000 7f0094b73200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:41:27.236 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:27 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:27.227+0000 7ff0d3c1a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:41:27.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:26 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/3687424477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:41:27.295 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:26 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:26.839+0000 7f99305ad200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:41:27.295 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:26 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:26.969+0000 7f99305ad200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:41:27.536 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:27 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:27.368+0000 7ff0d3c1a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:41:27.569 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-14T17:41:27.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-14T17:41:27.581 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:27.297+0000 7f99305ad200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:41:27.581 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:27.385+0000 7f99305ad200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:41:27.581 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:27.579+0000 7f99305ad200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:41:27.729 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:27 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:27.461+0000 7f0094b73200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:41:27.729 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:27 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:27.554+0000 7f0094b73200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:41:27.824 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:41:27.826 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:27 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:27.722+0000 7ff0d3c1a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:41:28.044 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:27.762+0000 7f99305ad200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:41:28.080 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:27 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:27.825+0000 7ff0d3c1a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:41:28.080 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:28 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:28.035+0000 7ff0d3c1a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:41:28.228 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:27 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:27.753+0000 7f0094b73200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:41:28.229 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:27 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:27.943+0000 7f0094b73200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:41:28.295 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:28 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:28.042+0000 7f99305ad200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:41:28.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:28 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:28.246+0000 7ff0d3c1a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:41:28.498 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:28 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:28.234+0000 7f0094b73200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:41:28.795 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:28 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:28.297+0000 7f99305ad200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:41:28.795 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:28 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:28.531+0000 7f99305ad200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:41:28.819 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:28 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:28.546+0000 7ff0d3c1a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:41:28.819 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:28 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:28.817+0000 7ff0d3c1a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:41:28.979 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:28 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:28.496+0000 7f0094b73200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:41:28.979 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:28 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:28.739+0000 7f0094b73200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:41:29.369 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:29 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:29.069+0000 7ff0d3c1a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:41:29.689 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:41:29.689 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"fsid":"932ae096-cb5f-11ee-95b9-87774f69a715","created":"2024-02-14T17:37:05.363563+0000","modified":"2024-02-14T17:41:09.396347+0000","last_up_change":"2024-02-14T17:40:47.973379+0000","last_in_change":"2024-02-14T17:40:28.275124+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-14T17:40:48.885325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-14T17:41:00.605527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-02-14T17:41:02.879326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-02-14T17:41:04.830834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-02-14T17:41:06.900998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"90eb8397-c65b-458e-a9fe-16e8b137a2ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1313296731},{"type":"v1","addr":"172.21.15.27:6803","nonce":1313296731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1313296731},{"type":"v1","addr":"172.21.15.27:6805","nonce":1313296731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1313296731},{"type":"v1","addr":"172.21.15.27:6809","nonce":1313296731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1313296731},{"type":"v1","addr":"172.21.15.27:6807","nonce":1313296731}]},"public_addr":"172.21.15.27:6803/1313296731","cluster_addr":"172.21.15.27:6805/1313296731","heartbeat_back_addr":"172.21.15.27:6809/1313296731","heartbeat_front_addr":"172.21.15.27:6807/1313296731","state":["exists","up"]},{"osd":1,"uuid":"a33d7701-d5f3-482e-8e5a-d658508b5468","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":4075500575},{"type":"v1","addr":"172.21.15.40:6801","nonce":4075500575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":4075500575},{"type":"v1","addr":"172.21.15.40:6803","nonce":4075500575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":4075500575},{"type":"v1","addr":"172.21.15.40:6807","nonce":4075500575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":4075500575},{"type":"v1","addr":"172.21.15.40:6805","nonce":4075500575}]},"public_addr":"172.21.15.40:6801/4075500575","cluster_addr":"172.21.15.40:6803/4075500575","heartbeat_back_addr":"172.21.15.40:6807/4075500575","heartbeat_front_addr":"172.21.15.40:6805/4075500575","state":["exists","up"]},{"osd":2,"uuid":"eac09c38-7d9b-43e1-8e02-7b99103304c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":1178359084},{"type":"v1","addr":"172.21.15.96:6801","nonce":1178359084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":1178359084},{"type":"v1","addr":"172.21.15.96:6803","nonce":1178359084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":1178359084},{"type":"v1","addr":"172.21.15.96:6807","nonce":1178359084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":1178359084},{"type":"v1","addr":"172.21.15.96:6805","nonce":1178359084}]},"public_addr":"172.21.15.96:6801/1178359084","cluster_addr":"172.21.15.96:6803/1178359084","heartbeat_back_addr":"172.21.15.96:6807/1178359084","heartbeat_front_addr":"172.21.15.96:6805/1178359084","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:39:55.426038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:40:20.620212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:40:45.953292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/681803954":"2024-02-15T17:38:12.353293+0000","172.21.15.27:0/3655809749":"2024-02-15T17:38:12.353293+0000","172.21.15.27:6801/2861113236":"2024-02-15T17:38:12.353293+0000","172.21.15.27:6800/2861113236":"2024-02-15T17:38:12.353293+0000","172.21.15.27:6801/4072918299":"2024-02-15T17:37:44.165328+0000","172.21.15.27:0/1727691481":"2024-02-15T17:37:44.165328+0000","172.21.15.27:6800/4072918299":"2024-02-15T17:37:44.165328+0000","172.21.15.27:0/3104063072":"2024-02-15T17:37:44.165328+0000","172.21.15.27:0/801052348":"2024-02-15T17:37:44.165328+0000","172.21.15.27:0/2483876909":"2024-02-15T17:38:12.353293+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-14T17:41:29.740 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:29 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:29.329+0000 7f99305ad200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:41:29.978 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:29 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/2577191561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:41:29.979 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:29 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:29.566+0000 7f0094b73200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:41:29.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:29 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/2577191561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:41:29.983 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:29 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:29.937+0000 7ff0d3c1a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:41:29.998 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:29 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/2577191561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:41:29.998 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:29 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:29.899+0000 7f99305ad200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:41:29.998 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:29 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:29.996+0000 7f99305ad200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:41:30.295 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:30.210+0000 7f99305ad200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:41:30.295 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: [14/Feb/2024:17:41:30] ENGINE Bus STARTING 2024-02-14T17:41:30.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-14T17:41:30.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-14T17:41:30.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-14T17:41:30.473 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:30.150+0000 7f0094b73200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:41:30.473 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:30.250+0000 7f0094b73200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:41:30.473 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:30.471+0000 7f0094b73200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:41:30.613 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:41:30.618 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:41:30.659 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:30.657+0000 7ff0d3c1a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:41:30.729 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: [14/Feb/2024:17:41:30] ENGINE Bus STARTING 2024-02-14T17:41:30.729 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: CherryPy Checker: 2024-02-14T17:41:30.729 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: The Application mounted at '' has an empty config. 2024-02-14T17:41:30.729 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: 2024-02-14T17:41:30.729 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: [14/Feb/2024:17:41:30] ENGINE Serving on http://:::9283 2024-02-14T17:41:30.729 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x[29242]: [14/Feb/2024:17:41:30] ENGINE Bus STARTED 2024-02-14T17:41:30.737 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: CherryPy Checker: 2024-02-14T17:41:30.737 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: The Application mounted at '' has an empty config. 2024-02-14T17:41:30.737 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: 2024-02-14T17:41:30.737 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: [14/Feb/2024:17:41:30] ENGINE Serving on http://:::9283 2024-02-14T17:41:30.738 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z[30095]: [14/Feb/2024:17:41:30] ENGINE Bus STARTED 2024-02-14T17:41:30.915 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:30.768+0000 7ff0d3c1a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:41:30.916 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-mon[25828]: Standby manager daemon z restarted 2024-02-14T17:41:30.916 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-mon[25828]: Standby manager daemon z started 2024-02-14T17:41:30.916 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.96:0/1045971408' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-14T17:41:30.916 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.96:0/1045971408' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:41:30.916 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.96:0/1045971408' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-14T17:41:30.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.96:0/1045971408' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:41:30.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-mon[25828]: Standby manager daemon x restarted 2024-02-14T17:41:30.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-mon[25828]: Standby manager daemon x started 2024-02-14T17:41:30.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.40:0/186253261' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:41:30.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.40:0/186253261' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:41:30.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.40:0/186253261' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:41:30.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:30 smithi027 ceph-mon[25828]: from='mgr.? 172.21.15.40:0/186253261' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:41:31.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-mon[27796]: Standby manager daemon z restarted 2024-02-14T17:41:31.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-mon[27796]: Standby manager daemon z started 2024-02-14T17:41:31.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.96:0/1045971408' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-14T17:41:31.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.96:0/1045971408' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:41:31.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.96:0/1045971408' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-14T17:41:31.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.96:0/1045971408' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:41:31.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-mon[27796]: Standby manager daemon x restarted 2024-02-14T17:41:31.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-mon[27796]: Standby manager daemon x started 2024-02-14T17:41:31.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.40:0/186253261' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:41:31.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.40:0/186253261' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:41:31.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.40:0/186253261' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:41:31.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:30 smithi096 ceph-mon[27796]: from='mgr.? 172.21.15.40:0/186253261' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:41:31.167 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:31.002+0000 7ff0d3c1a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:41:31.167 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:41:31] ENGINE Bus STARTING 2024-02-14T17:41:31.204 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-mon[27571]: Standby manager daemon z restarted 2024-02-14T17:41:31.204 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-mon[27571]: Standby manager daemon z started 2024-02-14T17:41:31.204 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.96:0/1045971408' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-14T17:41:31.204 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.96:0/1045971408' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:41:31.205 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.96:0/1045971408' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-14T17:41:31.205 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.96:0/1045971408' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:41:31.205 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-mon[27571]: Standby manager daemon x restarted 2024-02-14T17:41:31.205 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-mon[27571]: Standby manager daemon x started 2024-02-14T17:41:31.205 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.40:0/186253261' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:41:31.205 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.40:0/186253261' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:41:31.205 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.40:0/186253261' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:41:31.205 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:30 smithi040 ceph-mon[27571]: from='mgr.? 172.21.15.40:0/186253261' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:41:31.463 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: CherryPy Checker: 2024-02-14T17:41:31.463 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: The Application mounted at '' has an empty config. 2024-02-14T17:41:31.463 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:41:31.463 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:41:31] ENGINE Serving on http://:::9283 2024-02-14T17:41:31.463 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:41:31] ENGINE Bus STARTED 2024-02-14T17:41:32.045 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: Active manager daemon y restarted 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: Activating manager daemon y 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: osdmap e31: 3 total, 3 up, 3 in 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: mgrmap e19: y(active, starting, since 0.0133075s), standbys: x, z 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:41:32.046 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: Manager daemon y is now available 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]': finished 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]': finished 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]': finished 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:32.047 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:32.048 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: Updating smithi027:/etc/ceph/ceph.conf 2024-02-14T17:41:32.048 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: Updating smithi040:/etc/ceph/ceph.conf 2024-02-14T17:41:32.048 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:32.048 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: Updating smithi096:/etc/ceph/ceph.conf 2024-02-14T17:41:32.048 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:41:32.048 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:31 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:41:32.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-02-14T17:41:32.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: Active manager daemon y restarted 2024-02-14T17:41:32.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: Activating manager daemon y 2024-02-14T17:41:32.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: osdmap e31: 3 total, 3 up, 3 in 2024-02-14T17:41:32.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: mgrmap e19: y(active, starting, since 0.0133075s), standbys: x, z 2024-02-14T17:41:32.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:41:32.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:41:32.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:41:32.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:41:32.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:41:32.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-14T17:41:32.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:41:32.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:41:32.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:41:32.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: Manager daemon y is now available 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]': finished 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]': finished 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]': finished 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:32.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:32.223 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: Updating smithi027:/etc/ceph/ceph.conf 2024-02-14T17:41:32.223 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: Updating smithi040:/etc/ceph/ceph.conf 2024-02-14T17:41:32.223 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:32.223 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: Updating smithi096:/etc/ceph/ceph.conf 2024-02-14T17:41:32.223 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:41:32.223 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:31 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:41:32.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-02-14T17:41:32.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: Active manager daemon y restarted 2024-02-14T17:41:32.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: Activating manager daemon y 2024-02-14T17:41:32.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: osdmap e31: 3 total, 3 up, 3 in 2024-02-14T17:41:32.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: mgrmap e19: y(active, starting, since 0.0133075s), standbys: x, z 2024-02-14T17:41:32.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:41:32.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:41:32.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:41:32.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:41:32.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:41:32.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-14T17:41:32.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:41:32.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:41:32.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:41:32.231 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:41:32.231 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:41:32.231 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:41:32.231 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: Manager daemon y is now available 2024-02-14T17:41:32.231 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:32.231 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:32.231 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:32.231 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:41:32.231 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:41:32.231 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:41:32.231 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]': finished 2024-02-14T17:41:32.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]': finished 2024-02-14T17:41:32.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]': finished 2024-02-14T17:41:32.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:32.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:32.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:32.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: Updating smithi027:/etc/ceph/ceph.conf 2024-02-14T17:41:32.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: Updating smithi040:/etc/ceph/ceph.conf 2024-02-14T17:41:32.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:32.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: Updating smithi096:/etc/ceph/ceph.conf 2024-02-14T17:41:32.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:41:32.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:31 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:41:33.247 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: [14/Feb/2024:17:41:31] ENGINE Bus STARTING 2024-02-14T17:41:33.247 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: Updating smithi027:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:33.247 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:33.247 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:33.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: [14/Feb/2024:17:41:31] ENGINE Serving on https://172.21.15.27:7151 2024-02-14T17:41:33.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: [14/Feb/2024:17:41:31] ENGINE Serving on http://172.21.15.27:8765 2024-02-14T17:41:33.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: [14/Feb/2024:17:41:31] ENGINE Bus STARTED 2024-02-14T17:41:33.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:33.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:33.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:33.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: mgrmap e20: y(active, since 1.01528s), standbys: x, z 2024-02-14T17:41:33.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:33.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: Updating smithi027:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:33.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:33.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:33.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:41:33.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi027", "caps": []}]: dispatch 2024-02-14T17:41:33.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: [14/Feb/2024:17:41:31] ENGINE Bus STARTING 2024-02-14T17:41:33.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: Updating smithi027:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:33.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:33.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:33.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: [14/Feb/2024:17:41:31] ENGINE Serving on https://172.21.15.27:7151 2024-02-14T17:41:33.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: [14/Feb/2024:17:41:31] ENGINE Serving on http://172.21.15.27:8765 2024-02-14T17:41:33.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: [14/Feb/2024:17:41:31] ENGINE Bus STARTED 2024-02-14T17:41:33.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:33.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:33.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:33.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: mgrmap e20: y(active, since 1.01528s), standbys: x, z 2024-02-14T17:41:33.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:33.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: Updating smithi027:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:33.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:33.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:33.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.297 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.297 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.297 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.297 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:41:33.297 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.297 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi027", "caps": []}]: dispatch 2024-02-14T17:41:33.374 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: [14/Feb/2024:17:41:31] ENGINE Bus STARTING 2024-02-14T17:41:33.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: Updating smithi027:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:33.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:33.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:33.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: [14/Feb/2024:17:41:31] ENGINE Serving on https://172.21.15.27:7151 2024-02-14T17:41:33.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: [14/Feb/2024:17:41:31] ENGINE Serving on http://172.21.15.27:8765 2024-02-14T17:41:33.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: [14/Feb/2024:17:41:31] ENGINE Bus STARTED 2024-02-14T17:41:33.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:33.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:33.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:33.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: mgrmap e20: y(active, since 1.01528s), standbys: x, z 2024-02-14T17:41:33.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:33.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: Updating smithi027:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:33.376 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: Updating smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:33.377 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: Updating smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:33.377 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.377 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.377 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.377 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.377 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.377 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.377 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.377 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:41:33.377 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:33.377 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi027", "caps": []}]: dispatch 2024-02-14T17:41:34.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:34 smithi040 ceph-mon[27571]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:34.232 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:34 smithi027 ceph-mon[25828]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:34.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:34 smithi096 ceph-mon[27796]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:34.576 INFO:teuthology.orchestra.run.smithi027.stdout:77309411339 2024-02-14T17:41:34.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-14T17:41:34.829 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:41:35.212 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:35 smithi027 ceph-mon[25828]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-02-14T17:41:35.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:35 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi040", "caps": []}]: dispatch 2024-02-14T17:41:35.243 INFO:teuthology.orchestra.run.smithi027.stdout:55834574865 2024-02-14T17:41:35.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-14T17:41:35.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:35 smithi096 ceph-mon[27796]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-02-14T17:41:35.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:35 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi040", "caps": []}]: dispatch 2024-02-14T17:41:35.303 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:41:35.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:35 smithi040 ceph-mon[27571]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-02-14T17:41:35.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:35 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi040", "caps": []}]: dispatch 2024-02-14T17:41:36.334 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:36 smithi027 ceph-mon[25828]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-02-14T17:41:36.334 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:36 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi096", "caps": []}]: dispatch 2024-02-14T17:41:36.334 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:36 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:36.334 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:36 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:36.334 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:36 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:36.335 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:36 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: ::ffff:172.21.15.40 - - [14/Feb/2024:17:41:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-14T17:41:36.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:36 smithi040 ceph-mon[27571]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-02-14T17:41:36.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:36 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi096", "caps": []}]: dispatch 2024-02-14T17:41:36.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:36 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:36.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:36 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:36.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:36 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:36.462 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:36 smithi096 ceph-mon[27796]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-02-14T17:41:36.462 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:36 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi096", "caps": []}]: dispatch 2024-02-14T17:41:36.462 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:36 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:36.462 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:36 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:36.462 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:36 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:37 smithi096 ceph-mon[27796]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-02-14T17:41:37.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:37 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:37.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:37 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi096", "caps": []}]: dispatch 2024-02-14T17:41:37.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:37 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi040", "caps": []}]: dispatch 2024-02-14T17:41:37.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:37 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:37 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:37 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:37 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:37 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.296 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:37 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:37 smithi040 ceph-mon[27571]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-02-14T17:41:37.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:37 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:37.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:37 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi096", "caps": []}]: dispatch 2024-02-14T17:41:37.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:37 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi040", "caps": []}]: dispatch 2024-02-14T17:41:37.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:37 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:37 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:37 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:37 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:37 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:37 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:37 smithi027 ceph-mon[25828]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-02-14T17:41:37.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:37 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:37.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:37 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi096", "caps": []}]: dispatch 2024-02-14T17:41:37.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:37 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi040", "caps": []}]: dispatch 2024-02-14T17:41:37.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:37 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:37 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:37 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:37 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:37 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:37 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:37.506 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:41:38.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-14T17:41:38.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.229 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:38.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:38.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:41:38.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.230 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:38 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.460 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-14T17:41:38.460 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.460 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.460 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.460 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.460 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.460 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.460 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:38.461 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:38.461 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.461 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:41:38.461 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.461 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.461 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.461 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:38 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-14T17:41:38.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:38.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:38.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:41:38.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:38.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:38 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.441 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:39 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.441 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:39 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.442 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:39 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.442 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:39 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.442 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:39 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.442 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:39 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.442 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:39 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.442 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:39 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.442 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:39 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.442 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:39 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.442 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:39 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:39 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:39 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:39 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:39 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:39 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:39 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:39 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:39 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:39 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:39 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:39 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.583 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:39 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.583 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:39 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.583 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:39 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.583 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:39 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.583 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:39 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.583 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:39 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.583 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:39 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.584 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:39 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.584 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:39 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.584 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:39 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.584 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:39 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:39.730 INFO:teuthology.orchestra.run.smithi027.stdout:77309411340 2024-02-14T17:41:40.388 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-14T17:41:40.389 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.389 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.389 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.389 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.389 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.389 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/2832446060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-14T17:41:40.389 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.389 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.389 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.389 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.389 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.389 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.390 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:40 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.447 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-14T17:41:40.447 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.447 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.447 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.447 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.447 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.447 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/2832446060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-14T17:41:40.447 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.448 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.448 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.448 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.448 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.448 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.448 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:40 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.579 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-14T17:41:40.579 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.579 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.579 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.579 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.579 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.579 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/2832446060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-14T17:41:40.579 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.579 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.579 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.580 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.580 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.580 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:40.580 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:40 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:41.880 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411339 got 77309411340 for osd.2 2024-02-14T17:41:41.881 DEBUG:teuthology.parallel:result is None 2024-02-14T17:41:42.279 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-14T17:41:42.279 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.279 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.279 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.279 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.279 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:42.279 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:42.279 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:42.280 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.280 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: Deploying daemon alertmanager.a on smithi027 2024-02-14T17:41:42.280 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.280 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.280 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.280 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.280 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.280 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:42 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.298 INFO:teuthology.orchestra.run.smithi027.stdout:34359738391 2024-02-14T17:41:42.298 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-14T17:41:42.435 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-14T17:41:42.435 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.435 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.435 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.435 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.435 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:42.436 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:42.436 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:42.436 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.436 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: Deploying daemon alertmanager.a on smithi027 2024-02-14T17:41:42.436 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.436 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.436 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.436 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.436 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.436 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:42 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-14T17:41:42.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: Deploying daemon alertmanager.a on smithi027 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:42.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:42 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:43.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:43 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:43.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:43 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:43.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:43 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:43.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:43 smithi040 ceph-mon[27571]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:41:44.102 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:43 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:44.102 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:43 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:44.102 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:43 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:44.102 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:43 smithi027 ceph-mon[25828]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:41:44.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:43 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:44.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:43 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:44.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:43 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:44.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:43 smithi096 ceph-mon[27796]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:41:44.437 INFO:teuthology.orchestra.run.smithi027.stdout:55834574867 2024-02-14T17:41:44.799 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:41:44.899 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:44 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/1005853629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-14T17:41:45.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:44 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/1005853629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-14T17:41:45.270 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574865 got 55834574867 for osd.1 2024-02-14T17:41:45.270 DEBUG:teuthology.parallel:result is None 2024-02-14T17:41:45.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:44 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/1005853629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-14T17:41:46.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:46 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:46.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:46 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:46.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:46 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:46.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:46 smithi027 ceph-mon[25828]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-14T17:41:46.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:46 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:46.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:46 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:46.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:46 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:46.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:46 smithi096 ceph-mon[27796]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-14T17:41:46.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:46 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:46.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:46 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:46.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:46 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:46.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:46 smithi040 ceph-mon[27571]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-14T17:41:46.725 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:46 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: ::ffff:172.21.15.40 - - [14/Feb/2024:17:41:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-14T17:41:47.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:47 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:47.422 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:47 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:47.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:47 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:47.796 INFO:teuthology.orchestra.run.smithi027.stdout:34359738392 2024-02-14T17:41:48.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:48 smithi027 ceph-mon[25828]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:48.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:48 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/3828188668' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-14T17:41:48.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:48 smithi096 ceph-mon[27796]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:48.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:48 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/3828188668' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-14T17:41:48.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:48 smithi040 ceph-mon[27571]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:48.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:48 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/3828188668' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-14T17:41:49.242 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738391 got 34359738392 for osd.0 2024-02-14T17:41:49.243 DEBUG:teuthology.parallel:result is None 2024-02-14T17:41:49.243 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-14T17:41:49.243 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-14T17:41:50.189 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:50 smithi027 ceph-mon[25828]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:41:50.480 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:50 smithi040 ceph-mon[27571]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:50.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:50 smithi096 ceph-mon[27796]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:52.109 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:52 smithi027 ceph-mon[25828]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:52.347 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:52 smithi096 ceph-mon[27796]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:52.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:52 smithi040 ceph-mon[27571]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:54.357 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:54 smithi027 ceph-mon[25828]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:54.357 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:54 smithi027 podman[48714]: 2024-02-14 17:41:54.343220101 +0000 UTC m=+0.102811856 volume create ea436f0a3d0bfa46b6b03dfebb66a6ae67e6208153530c78cd27f6644891b34d 2024-02-14T17:41:54.357 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:54 smithi027 podman[48714]: 2024-02-14 17:41:54.261291567 +0000 UTC m=+0.020883351 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-14T17:41:54.389 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:41:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-02-14T17:41:54.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:54 smithi040 ceph-mon[27571]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:54.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:54 smithi096 ceph-mon[27796]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:54.629 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:54 smithi027 podman[48714]: 2024-02-14 17:41:54.452411602 +0000 UTC m=+0.212003401 container create c74a7ffc7aa9e13c044b7202524736aaf9fc6c7d8a3f84d187652cc52fcd0a6b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:55.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:55 smithi027 ceph-mon[25828]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:41:55.218 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:54 smithi027 podman[48714]: 2024-02-14 17:41:54.947409491 +0000 UTC m=+0.707001247 container init c74a7ffc7aa9e13c044b7202524736aaf9fc6c7d8a3f84d187652cc52fcd0a6b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:55.219 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:54 smithi027 podman[48714]: 2024-02-14 17:41:54.953526857 +0000 UTC m=+0.713118613 container start c74a7ffc7aa9e13c044b7202524736aaf9fc6c7d8a3f84d187652cc52fcd0a6b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:55.219 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:54 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[48745]: ts=2024-02-14T17:41:54.967Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-14T17:41:55.219 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:54 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[48745]: ts=2024-02-14T17:41:54.967Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-14T17:41:55.219 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:54 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[48745]: ts=2024-02-14T17:41:54.968Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.27 port=9094 2024-02-14T17:41:55.219 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:54 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[48745]: ts=2024-02-14T17:41:54.969Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-14T17:41:55.219 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:54 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[48745]: ts=2024-02-14T17:41:54.992Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-14T17:41:55.219 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:54 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[48745]: ts=2024-02-14T17:41:54.992Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-14T17:41:55.219 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:54 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[48745]: ts=2024-02-14T17:41:54.995Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-14T17:41:55.219 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:54 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[48745]: ts=2024-02-14T17:41:54.995Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-14T17:41:55.219 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:55 smithi027 bash[48714]: c74a7ffc7aa9e13c044b7202524736aaf9fc6c7d8a3f84d187652cc52fcd0a6b 2024-02-14T17:41:55.219 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:55 smithi027 systemd[1]: Started Ceph alertmanager.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:41:55.380 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:55 smithi040 ceph-mon[27571]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:41:55.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:55 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:41:55] ENGINE Bus STOPPING 2024-02-14T17:41:55.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:55 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:41:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-14T17:41:55.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:55 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:41:55] ENGINE Bus STOPPED 2024-02-14T17:41:55.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:55 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:41:55] ENGINE Bus STARTING 2024-02-14T17:41:55.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:55 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:41:55] ENGINE Serving on http://:::9283 2024-02-14T17:41:55.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:55 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:41:55] ENGINE Bus STARTED 2024-02-14T17:41:55.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:55 smithi096 ceph-mon[27796]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:41:55.650 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-02-14T17:41:53.039162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96620,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126548,"statfs":{"total":287972524032,"available":287873585152,"internally_reserved":0,"allocated":3158016,"data_stored":2100345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002957"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023532+0000","last_change":"2024-02-14T17:41:08.409572+0000","last_active":"2024-02-14T17:41:31.023532+0000","last_peered":"2024-02-14T17:41:31.023532+0000","last_clean":"2024-02-14T17:41:31.023532+0000","last_became_active":"2024-02-14T17:41:08.409197+0000","last_became_peered":"2024-02-14T17:41:08.409197+0000","last_unstale":"2024-02-14T17:41:31.023532+0000","last_undegraded":"2024-02-14T17:41:31.023532+0000","last_fullsized":"2024-02-14T17:41:31.023532+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:52:18.272045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024775+0000","last_change":"2024-02-14T17:41:06.398484+0000","last_active":"2024-02-14T17:41:31.024775+0000","last_peered":"2024-02-14T17:41:31.024775+0000","last_clean":"2024-02-14T17:41:31.024775+0000","last_became_active":"2024-02-14T17:41:06.397773+0000","last_became_peered":"2024-02-14T17:41:06.397773+0000","last_unstale":"2024-02-14T17:41:31.024775+0000","last_undegraded":"2024-02-14T17:41:31.024775+0000","last_fullsized":"2024-02-14T17:41:31.024775+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:44:45.584250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027399+0000","last_change":"2024-02-14T17:41:02.379405+0000","last_active":"2024-02-14T17:41:31.027399+0000","last_peered":"2024-02-14T17:41:31.027399+0000","last_clean":"2024-02-14T17:41:31.027399+0000","last_became_active":"2024-02-14T17:41:02.379088+0000","last_became_peered":"2024-02-14T17:41:02.379088+0000","last_unstale":"2024-02-14T17:41:31.027399+0000","last_undegraded":"2024-02-14T17:41:31.027399+0000","last_fullsized":"2024-02-14T17:41:31.027399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:41:52.534467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024894+0000","last_change":"2024-02-14T17:41:04.384996+0000","last_active":"2024-02-14T17:41:31.024894+0000","last_peered":"2024-02-14T17:41:31.024894+0000","last_clean":"2024-02-14T17:41:31.024894+0000","last_became_active":"2024-02-14T17:41:04.384539+0000","last_became_peered":"2024-02-14T17:41:04.384539+0000","last_unstale":"2024-02-14T17:41:31.024894+0000","last_undegraded":"2024-02-14T17:41:31.024894+0000","last_fullsized":"2024-02-14T17:41:31.024894+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:36:56.773302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023725+0000","last_change":"2024-02-14T17:41:08.408347+0000","last_active":"2024-02-14T17:41:31.023725+0000","last_peered":"2024-02-14T17:41:31.023725+0000","last_clean":"2024-02-14T17:41:31.023725+0000","last_became_active":"2024-02-14T17:41:08.407865+0000","last_became_peered":"2024-02-14T17:41:08.407865+0000","last_unstale":"2024-02-14T17:41:31.023725+0000","last_undegraded":"2024-02-14T17:41:31.023725+0000","last_fullsized":"2024-02-14T17:41:31.023725+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:12:19.847910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023431+0000","last_change":"2024-02-14T17:41:06.401682+0000","last_active":"2024-02-14T17:41:31.023431+0000","last_peered":"2024-02-14T17:41:31.023431+0000","last_clean":"2024-02-14T17:41:31.023431+0000","last_became_active":"2024-02-14T17:41:06.400268+0000","last_became_peered":"2024-02-14T17:41:06.400268+0000","last_unstale":"2024-02-14T17:41:31.023431+0000","last_undegraded":"2024-02-14T17:41:31.023431+0000","last_fullsized":"2024-02-14T17:41:31.023431+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:24:15.778505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025322+0000","last_change":"2024-02-14T17:41:02.378302+0000","last_active":"2024-02-14T17:41:31.025322+0000","last_peered":"2024-02-14T17:41:31.025322+0000","last_clean":"2024-02-14T17:41:31.025322+0000","last_became_active":"2024-02-14T17:41:02.377857+0000","last_became_peered":"2024-02-14T17:41:02.377857+0000","last_unstale":"2024-02-14T17:41:31.025322+0000","last_undegraded":"2024-02-14T17:41:31.025322+0000","last_fullsized":"2024-02-14T17:41:31.025322+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:39:11.805991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026496+0000","last_change":"2024-02-14T17:41:04.386956+0000","last_active":"2024-02-14T17:41:31.026496+0000","last_peered":"2024-02-14T17:41:31.026496+0000","last_clean":"2024-02-14T17:41:31.026496+0000","last_became_active":"2024-02-14T17:41:04.386184+0000","last_became_peered":"2024-02-14T17:41:04.386184+0000","last_unstale":"2024-02-14T17:41:31.026496+0000","last_undegraded":"2024-02-14T17:41:31.026496+0000","last_fullsized":"2024-02-14T17:41:31.026496+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:37:54.980130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024842+0000","last_change":"2024-02-14T17:41:08.412583+0000","last_active":"2024-02-14T17:41:31.024842+0000","last_peered":"2024-02-14T17:41:31.024842+0000","last_clean":"2024-02-14T17:41:31.024842+0000","last_became_active":"2024-02-14T17:41:08.412286+0000","last_became_peered":"2024-02-14T17:41:08.412286+0000","last_unstale":"2024-02-14T17:41:31.024842+0000","last_undegraded":"2024-02-14T17:41:31.024842+0000","last_fullsized":"2024-02-14T17:41:31.024842+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:56:11.894166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024273+0000","last_change":"2024-02-14T17:41:06.400846+0000","last_active":"2024-02-14T17:41:31.024273+0000","last_peered":"2024-02-14T17:41:31.024273+0000","last_clean":"2024-02-14T17:41:31.024273+0000","last_became_active":"2024-02-14T17:41:06.400500+0000","last_became_peered":"2024-02-14T17:41:06.400500+0000","last_unstale":"2024-02-14T17:41:31.024273+0000","last_undegraded":"2024-02-14T17:41:31.024273+0000","last_fullsized":"2024-02-14T17:41:31.024273+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:44:46.320226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027855+0000","last_change":"2024-02-14T17:41:02.379380+0000","last_active":"2024-02-14T17:41:31.027855+0000","last_peered":"2024-02-14T17:41:31.027855+0000","last_clean":"2024-02-14T17:41:31.027855+0000","last_became_active":"2024-02-14T17:41:02.378234+0000","last_became_peered":"2024-02-14T17:41:02.378234+0000","last_unstale":"2024-02-14T17:41:31.027855+0000","last_undegraded":"2024-02-14T17:41:31.027855+0000","last_fullsized":"2024-02-14T17:41:31.027855+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:36:35.084518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026134+0000","last_change":"2024-02-14T17:41:04.386399+0000","last_active":"2024-02-14T17:41:31.026134+0000","last_peered":"2024-02-14T17:41:31.026134+0000","last_clean":"2024-02-14T17:41:31.026134+0000","last_became_active":"2024-02-14T17:41:04.385166+0000","last_became_peered":"2024-02-14T17:41:04.385166+0000","last_unstale":"2024-02-14T17:41:31.026134+0000","last_undegraded":"2024-02-14T17:41:31.026134+0000","last_fullsized":"2024-02-14T17:41:31.026134+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:52:15.458766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024012+0000","last_change":"2024-02-14T17:41:08.412131+0000","last_active":"2024-02-14T17:41:31.024012+0000","last_peered":"2024-02-14T17:41:31.024012+0000","last_clean":"2024-02-14T17:41:31.024012+0000","last_became_active":"2024-02-14T17:41:08.411651+0000","last_became_peered":"2024-02-14T17:41:08.411651+0000","last_unstale":"2024-02-14T17:41:31.024012+0000","last_undegraded":"2024-02-14T17:41:31.024012+0000","last_fullsized":"2024-02-14T17:41:31.024012+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:34:35.224612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023387+0000","last_change":"2024-02-14T17:41:06.401047+0000","last_active":"2024-02-14T17:41:31.023387+0000","last_peered":"2024-02-14T17:41:31.023387+0000","last_clean":"2024-02-14T17:41:31.023387+0000","last_became_active":"2024-02-14T17:41:06.400216+0000","last_became_peered":"2024-02-14T17:41:06.400216+0000","last_unstale":"2024-02-14T17:41:31.023387+0000","last_undegraded":"2024-02-14T17:41:31.023387+0000","last_fullsized":"2024-02-14T17:41:31.023387+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:44:41.178492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026742+0000","last_change":"2024-02-14T17:41:02.373621+0000","last_active":"2024-02-14T17:41:31.026742+0000","last_peered":"2024-02-14T17:41:31.026742+0000","last_clean":"2024-02-14T17:41:31.026742+0000","last_became_active":"2024-02-14T17:41:02.373205+0000","last_became_peered":"2024-02-14T17:41:02.373205+0000","last_unstale":"2024-02-14T17:41:31.026742+0000","last_undegraded":"2024-02-14T17:41:31.026742+0000","last_fullsized":"2024-02-14T17:41:31.026742+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:01:22.802544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027564+0000","last_change":"2024-02-14T17:41:04.390936+0000","last_active":"2024-02-14T17:41:31.027564+0000","last_peered":"2024-02-14T17:41:31.027564+0000","last_clean":"2024-02-14T17:41:31.027564+0000","last_became_active":"2024-02-14T17:41:04.390581+0000","last_became_peered":"2024-02-14T17:41:04.390581+0000","last_unstale":"2024-02-14T17:41:31.027564+0000","last_undegraded":"2024-02-14T17:41:31.027564+0000","last_fullsized":"2024-02-14T17:41:31.027564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:15:05.100763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025762+0000","last_change":"2024-02-14T17:41:08.412238+0000","last_active":"2024-02-14T17:41:31.025762+0000","last_peered":"2024-02-14T17:41:31.025762+0000","last_clean":"2024-02-14T17:41:31.025762+0000","last_became_active":"2024-02-14T17:41:08.411851+0000","last_became_peered":"2024-02-14T17:41:08.411851+0000","last_unstale":"2024-02-14T17:41:31.025762+0000","last_undegraded":"2024-02-14T17:41:31.025762+0000","last_fullsized":"2024-02-14T17:41:31.025762+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:14:43.981796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026687+0000","last_change":"2024-02-14T17:41:06.397548+0000","last_active":"2024-02-14T17:41:31.026687+0000","last_peered":"2024-02-14T17:41:31.026687+0000","last_clean":"2024-02-14T17:41:31.026687+0000","last_became_active":"2024-02-14T17:41:06.397263+0000","last_became_peered":"2024-02-14T17:41:06.397263+0000","last_unstale":"2024-02-14T17:41:31.026687+0000","last_undegraded":"2024-02-14T17:41:31.026687+0000","last_fullsized":"2024-02-14T17:41:31.026687+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:20:47.373218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025730+0000","last_change":"2024-02-14T17:41:02.374687+0000","last_active":"2024-02-14T17:41:31.025730+0000","last_peered":"2024-02-14T17:41:31.025730+0000","last_clean":"2024-02-14T17:41:31.025730+0000","last_became_active":"2024-02-14T17:41:02.374361+0000","last_became_peered":"2024-02-14T17:41:02.374361+0000","last_unstale":"2024-02-14T17:41:31.025730+0000","last_undegraded":"2024-02-14T17:41:31.025730+0000","last_fullsized":"2024-02-14T17:41:31.025730+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:39:56.915137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025779+0000","last_change":"2024-02-14T17:41:04.391022+0000","last_active":"2024-02-14T17:41:31.025779+0000","last_peered":"2024-02-14T17:41:31.025779+0000","last_clean":"2024-02-14T17:41:31.025779+0000","last_became_active":"2024-02-14T17:41:04.390597+0000","last_became_peered":"2024-02-14T17:41:04.390597+0000","last_unstale":"2024-02-14T17:41:31.025779+0000","last_undegraded":"2024-02-14T17:41:31.025779+0000","last_fullsized":"2024-02-14T17:41:31.025779+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T00:11:27.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025501+0000","last_change":"2024-02-14T17:41:08.411065+0000","last_active":"2024-02-14T17:41:31.025501+0000","last_peered":"2024-02-14T17:41:31.025501+0000","last_clean":"2024-02-14T17:41:31.025501+0000","last_became_active":"2024-02-14T17:41:08.410761+0000","last_became_peered":"2024-02-14T17:41:08.410761+0000","last_unstale":"2024-02-14T17:41:31.025501+0000","last_undegraded":"2024-02-14T17:41:31.025501+0000","last_fullsized":"2024-02-14T17:41:31.025501+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:28:34.606698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026657+0000","last_change":"2024-02-14T17:41:06.397434+0000","last_active":"2024-02-14T17:41:31.026657+0000","last_peered":"2024-02-14T17:41:31.026657+0000","last_clean":"2024-02-14T17:41:31.026657+0000","last_became_active":"2024-02-14T17:41:06.397004+0000","last_became_peered":"2024-02-14T17:41:06.397004+0000","last_unstale":"2024-02-14T17:41:31.026657+0000","last_undegraded":"2024-02-14T17:41:31.026657+0000","last_fullsized":"2024-02-14T17:41:31.026657+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:22:26.898910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024052+0000","last_change":"2024-02-14T17:41:02.374878+0000","last_active":"2024-02-14T17:41:31.024052+0000","last_peered":"2024-02-14T17:41:31.024052+0000","last_clean":"2024-02-14T17:41:31.024052+0000","last_became_active":"2024-02-14T17:41:02.374387+0000","last_became_peered":"2024-02-14T17:41:02.374387+0000","last_unstale":"2024-02-14T17:41:31.024052+0000","last_undegraded":"2024-02-14T17:41:31.024052+0000","last_fullsized":"2024-02-14T17:41:31.024052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:12:05.740629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025525+0000","last_change":"2024-02-14T17:41:04.391265+0000","last_active":"2024-02-14T17:41:31.025525+0000","last_peered":"2024-02-14T17:41:31.025525+0000","last_clean":"2024-02-14T17:41:31.025525+0000","last_became_active":"2024-02-14T17:41:04.390926+0000","last_became_peered":"2024-02-14T17:41:04.390926+0000","last_unstale":"2024-02-14T17:41:31.025525+0000","last_undegraded":"2024-02-14T17:41:31.025525+0000","last_fullsized":"2024-02-14T17:41:31.025525+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T00:22:23.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023953+0000","last_change":"2024-02-14T17:41:08.412098+0000","last_active":"2024-02-14T17:41:31.023953+0000","last_peered":"2024-02-14T17:41:31.023953+0000","last_clean":"2024-02-14T17:41:31.023953+0000","last_became_active":"2024-02-14T17:41:08.411698+0000","last_became_peered":"2024-02-14T17:41:08.411698+0000","last_unstale":"2024-02-14T17:41:31.023953+0000","last_undegraded":"2024-02-14T17:41:31.023953+0000","last_fullsized":"2024-02-14T17:41:31.023953+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:39:45.814581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:42.426382+0000","last_change":"2024-02-14T17:41:06.400575+0000","last_active":"2024-02-14T17:41:42.426382+0000","last_peered":"2024-02-14T17:41:42.426382+0000","last_clean":"2024-02-14T17:41:42.426382+0000","last_became_active":"2024-02-14T17:41:06.399708+0000","last_became_peered":"2024-02-14T17:41:06.399708+0000","last_unstale":"2024-02-14T17:41:42.426382+0000","last_undegraded":"2024-02-14T17:41:42.426382+0000","last_fullsized":"2024-02-14T17:41:42.426382+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:44:25.603955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024191+0000","last_change":"2024-02-14T17:41:02.375352+0000","last_active":"2024-02-14T17:41:31.024191+0000","last_peered":"2024-02-14T17:41:31.024191+0000","last_clean":"2024-02-14T17:41:31.024191+0000","last_became_active":"2024-02-14T17:41:02.374866+0000","last_became_peered":"2024-02-14T17:41:02.374866+0000","last_unstale":"2024-02-14T17:41:31.024191+0000","last_undegraded":"2024-02-14T17:41:31.024191+0000","last_fullsized":"2024-02-14T17:41:31.024191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:24:12.343347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023132+0000","last_change":"2024-02-14T17:41:04.385887+0000","last_active":"2024-02-14T17:41:31.023132+0000","last_peered":"2024-02-14T17:41:31.023132+0000","last_clean":"2024-02-14T17:41:31.023132+0000","last_became_active":"2024-02-14T17:41:04.384679+0000","last_became_peered":"2024-02-14T17:41:04.384679+0000","last_unstale":"2024-02-14T17:41:31.023132+0000","last_undegraded":"2024-02-14T17:41:31.023132+0000","last_fullsized":"2024-02-14T17:41:31.023132+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:22:17.122776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024119+0000","last_change":"2024-02-14T17:41:08.412325+0000","last_active":"2024-02-14T17:41:31.024119+0000","last_peered":"2024-02-14T17:41:31.024119+0000","last_clean":"2024-02-14T17:41:31.024119+0000","last_became_active":"2024-02-14T17:41:08.411948+0000","last_became_peered":"2024-02-14T17:41:08.411948+0000","last_unstale":"2024-02-14T17:41:31.024119+0000","last_undegraded":"2024-02-14T17:41:31.024119+0000","last_fullsized":"2024-02-14T17:41:31.024119+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:11:08.186746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024303+0000","last_change":"2024-02-14T17:41:06.398714+0000","last_active":"2024-02-14T17:41:31.024303+0000","last_peered":"2024-02-14T17:41:31.024303+0000","last_clean":"2024-02-14T17:41:31.024303+0000","last_became_active":"2024-02-14T17:41:06.398188+0000","last_became_peered":"2024-02-14T17:41:06.398188+0000","last_unstale":"2024-02-14T17:41:31.024303+0000","last_undegraded":"2024-02-14T17:41:31.024303+0000","last_fullsized":"2024-02-14T17:41:31.024303+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:04:10.150694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024783+0000","last_change":"2024-02-14T17:41:02.374502+0000","last_active":"2024-02-14T17:41:31.024783+0000","last_peered":"2024-02-14T17:41:31.024783+0000","last_clean":"2024-02-14T17:41:31.024783+0000","last_became_active":"2024-02-14T17:41:02.373525+0000","last_became_peered":"2024-02-14T17:41:02.373525+0000","last_unstale":"2024-02-14T17:41:31.024783+0000","last_undegraded":"2024-02-14T17:41:31.024783+0000","last_fullsized":"2024-02-14T17:41:31.024783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:10:26.283469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024339+0000","last_change":"2024-02-14T17:41:04.384407+0000","last_active":"2024-02-14T17:41:31.024339+0000","last_peered":"2024-02-14T17:41:31.024339+0000","last_clean":"2024-02-14T17:41:31.024339+0000","last_became_active":"2024-02-14T17:41:04.384027+0000","last_became_peered":"2024-02-14T17:41:04.384027+0000","last_unstale":"2024-02-14T17:41:31.024339+0000","last_undegraded":"2024-02-14T17:41:31.024339+0000","last_fullsized":"2024-02-14T17:41:31.024339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:18:32.190114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023649+0000","last_change":"2024-02-14T17:41:08.409448+0000","last_active":"2024-02-14T17:41:31.023649+0000","last_peered":"2024-02-14T17:41:31.023649+0000","last_clean":"2024-02-14T17:41:31.023649+0000","last_became_active":"2024-02-14T17:41:08.408388+0000","last_became_peered":"2024-02-14T17:41:08.408388+0000","last_unstale":"2024-02-14T17:41:31.023649+0000","last_undegraded":"2024-02-14T17:41:31.023649+0000","last_fullsized":"2024-02-14T17:41:31.023649+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:11:32.120421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023388+0000","last_change":"2024-02-14T17:41:06.400935+0000","last_active":"2024-02-14T17:41:31.023388+0000","last_peered":"2024-02-14T17:41:31.023388+0000","last_clean":"2024-02-14T17:41:31.023388+0000","last_became_active":"2024-02-14T17:41:06.400550+0000","last_became_peered":"2024-02-14T17:41:06.400550+0000","last_unstale":"2024-02-14T17:41:31.023388+0000","last_undegraded":"2024-02-14T17:41:31.023388+0000","last_fullsized":"2024-02-14T17:41:31.023388+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:19:57.099758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025178+0000","last_change":"2024-02-14T17:41:02.377901+0000","last_active":"2024-02-14T17:41:31.025178+0000","last_peered":"2024-02-14T17:41:31.025178+0000","last_clean":"2024-02-14T17:41:31.025178+0000","last_became_active":"2024-02-14T17:41:02.377486+0000","last_became_peered":"2024-02-14T17:41:02.377486+0000","last_unstale":"2024-02-14T17:41:31.025178+0000","last_undegraded":"2024-02-14T17:41:31.025178+0000","last_fullsized":"2024-02-14T17:41:31.025178+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:47:16.432930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025655+0000","last_change":"2024-02-14T17:41:04.384956+0000","last_active":"2024-02-14T17:41:31.025655+0000","last_peered":"2024-02-14T17:41:31.025655+0000","last_clean":"2024-02-14T17:41:31.025655+0000","last_became_active":"2024-02-14T17:41:04.384575+0000","last_became_peered":"2024-02-14T17:41:04.384575+0000","last_unstale":"2024-02-14T17:41:31.025655+0000","last_undegraded":"2024-02-14T17:41:31.025655+0000","last_fullsized":"2024-02-14T17:41:31.025655+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:48:45.219212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023612+0000","last_change":"2024-02-14T17:41:08.406762+0000","last_active":"2024-02-14T17:41:31.023612+0000","last_peered":"2024-02-14T17:41:31.023612+0000","last_clean":"2024-02-14T17:41:31.023612+0000","last_became_active":"2024-02-14T17:41:08.406418+0000","last_became_peered":"2024-02-14T17:41:08.406418+0000","last_unstale":"2024-02-14T17:41:31.023612+0000","last_undegraded":"2024-02-14T17:41:31.023612+0000","last_fullsized":"2024-02-14T17:41:31.023612+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:11:54.601485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023477+0000","last_change":"2024-02-14T17:41:06.398956+0000","last_active":"2024-02-14T17:41:31.023477+0000","last_peered":"2024-02-14T17:41:31.023477+0000","last_clean":"2024-02-14T17:41:31.023477+0000","last_became_active":"2024-02-14T17:41:06.398597+0000","last_became_peered":"2024-02-14T17:41:06.398597+0000","last_unstale":"2024-02-14T17:41:31.023477+0000","last_undegraded":"2024-02-14T17:41:31.023477+0000","last_fullsized":"2024-02-14T17:41:31.023477+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T17:46:04.813583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025208+0000","last_change":"2024-02-14T17:41:02.376868+0000","last_active":"2024-02-14T17:41:31.025208+0000","last_peered":"2024-02-14T17:41:31.025208+0000","last_clean":"2024-02-14T17:41:31.025208+0000","last_became_active":"2024-02-14T17:41:02.375350+0000","last_became_peered":"2024-02-14T17:41:02.375350+0000","last_unstale":"2024-02-14T17:41:31.025208+0000","last_undegraded":"2024-02-14T17:41:31.025208+0000","last_fullsized":"2024-02-14T17:41:31.025208+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:54:07.670987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027287+0000","last_change":"2024-02-14T17:41:04.389522+0000","last_active":"2024-02-14T17:41:31.027287+0000","last_peered":"2024-02-14T17:41:31.027287+0000","last_clean":"2024-02-14T17:41:31.027287+0000","last_became_active":"2024-02-14T17:41:04.388836+0000","last_became_peered":"2024-02-14T17:41:04.388836+0000","last_unstale":"2024-02-14T17:41:31.027287+0000","last_undegraded":"2024-02-14T17:41:31.027287+0000","last_fullsized":"2024-02-14T17:41:31.027287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:18:14.679143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025075+0000","last_change":"2024-02-14T17:41:08.412607+0000","last_active":"2024-02-14T17:41:31.025075+0000","last_peered":"2024-02-14T17:41:31.025075+0000","last_clean":"2024-02-14T17:41:31.025075+0000","last_became_active":"2024-02-14T17:41:08.412347+0000","last_became_peered":"2024-02-14T17:41:08.412347+0000","last_unstale":"2024-02-14T17:41:31.025075+0000","last_undegraded":"2024-02-14T17:41:31.025075+0000","last_fullsized":"2024-02-14T17:41:31.025075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T17:58:04.627588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024457+0000","last_change":"2024-02-14T17:41:06.401845+0000","last_active":"2024-02-14T17:41:31.024457+0000","last_peered":"2024-02-14T17:41:31.024457+0000","last_clean":"2024-02-14T17:41:31.024457+0000","last_became_active":"2024-02-14T17:41:06.401342+0000","last_became_peered":"2024-02-14T17:41:06.401342+0000","last_unstale":"2024-02-14T17:41:31.024457+0000","last_undegraded":"2024-02-14T17:41:31.024457+0000","last_fullsized":"2024-02-14T17:41:31.024457+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:26:51.292967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027871+0000","last_change":"2024-02-14T17:41:02.378515+0000","last_active":"2024-02-14T17:41:31.027871+0000","last_peered":"2024-02-14T17:41:31.027871+0000","last_clean":"2024-02-14T17:41:31.027871+0000","last_became_active":"2024-02-14T17:41:02.377309+0000","last_became_peered":"2024-02-14T17:41:02.377309+0000","last_unstale":"2024-02-14T17:41:31.027871+0000","last_undegraded":"2024-02-14T17:41:31.027871+0000","last_fullsized":"2024-02-14T17:41:31.027871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:02:04.610514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026229+0000","last_change":"2024-02-14T17:41:04.387230+0000","last_active":"2024-02-14T17:41:31.026229+0000","last_peered":"2024-02-14T17:41:31.026229+0000","last_clean":"2024-02-14T17:41:31.026229+0000","last_became_active":"2024-02-14T17:41:04.386230+0000","last_became_peered":"2024-02-14T17:41:04.386230+0000","last_unstale":"2024-02-14T17:41:31.026229+0000","last_undegraded":"2024-02-14T17:41:31.026229+0000","last_fullsized":"2024-02-14T17:41:31.026229+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:28:25.522058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025123+0000","last_change":"2024-02-14T17:41:08.412479+0000","last_active":"2024-02-14T17:41:31.025123+0000","last_peered":"2024-02-14T17:41:31.025123+0000","last_clean":"2024-02-14T17:41:31.025123+0000","last_became_active":"2024-02-14T17:41:08.412046+0000","last_became_peered":"2024-02-14T17:41:08.412046+0000","last_unstale":"2024-02-14T17:41:31.025123+0000","last_undegraded":"2024-02-14T17:41:31.025123+0000","last_fullsized":"2024-02-14T17:41:31.025123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:23:58.892388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023499+0000","last_change":"2024-02-14T17:41:06.398121+0000","last_active":"2024-02-14T17:41:31.023499+0000","last_peered":"2024-02-14T17:41:31.023499+0000","last_clean":"2024-02-14T17:41:31.023499+0000","last_became_active":"2024-02-14T17:41:06.397728+0000","last_became_peered":"2024-02-14T17:41:06.397728+0000","last_unstale":"2024-02-14T17:41:31.023499+0000","last_undegraded":"2024-02-14T17:41:31.023499+0000","last_fullsized":"2024-02-14T17:41:31.023499+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:02:11.540347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025990+0000","last_change":"2024-02-14T17:41:02.376309+0000","last_active":"2024-02-14T17:41:31.025990+0000","last_peered":"2024-02-14T17:41:31.025990+0000","last_clean":"2024-02-14T17:41:31.025990+0000","last_became_active":"2024-02-14T17:41:02.374567+0000","last_became_peered":"2024-02-14T17:41:02.374567+0000","last_unstale":"2024-02-14T17:41:31.025990+0000","last_undegraded":"2024-02-14T17:41:31.025990+0000","last_fullsized":"2024-02-14T17:41:31.025990+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:20:07.899727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027571+0000","last_change":"2024-02-14T17:41:04.391275+0000","last_active":"2024-02-14T17:41:31.027571+0000","last_peered":"2024-02-14T17:41:31.027571+0000","last_clean":"2024-02-14T17:41:31.027571+0000","last_became_active":"2024-02-14T17:41:04.390816+0000","last_became_peered":"2024-02-14T17:41:04.390816+0000","last_unstale":"2024-02-14T17:41:31.027571+0000","last_undegraded":"2024-02-14T17:41:31.027571+0000","last_fullsized":"2024-02-14T17:41:31.027571+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:24:46.327420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025567+0000","last_change":"2024-02-14T17:41:08.409579+0000","last_active":"2024-02-14T17:41:31.025567+0000","last_peered":"2024-02-14T17:41:31.025567+0000","last_clean":"2024-02-14T17:41:31.025567+0000","last_became_active":"2024-02-14T17:41:08.409224+0000","last_became_peered":"2024-02-14T17:41:08.409224+0000","last_unstale":"2024-02-14T17:41:31.025567+0000","last_undegraded":"2024-02-14T17:41:31.025567+0000","last_fullsized":"2024-02-14T17:41:31.025567+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:06:13.387021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:42.426389+0000","last_change":"2024-02-14T17:41:06.400941+0000","last_active":"2024-02-14T17:41:42.426389+0000","last_peered":"2024-02-14T17:41:42.426389+0000","last_clean":"2024-02-14T17:41:42.426389+0000","last_became_active":"2024-02-14T17:41:06.399688+0000","last_became_peered":"2024-02-14T17:41:06.399688+0000","last_unstale":"2024-02-14T17:41:42.426389+0000","last_undegraded":"2024-02-14T17:41:42.426389+0000","last_fullsized":"2024-02-14T17:41:42.426389+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:12:16.887899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025837+0000","last_change":"2024-02-14T17:41:02.374652+0000","last_active":"2024-02-14T17:41:31.025837+0000","last_peered":"2024-02-14T17:41:31.025837+0000","last_clean":"2024-02-14T17:41:31.025837+0000","last_became_active":"2024-02-14T17:41:02.373637+0000","last_became_peered":"2024-02-14T17:41:02.373637+0000","last_unstale":"2024-02-14T17:41:31.025837+0000","last_undegraded":"2024-02-14T17:41:31.025837+0000","last_fullsized":"2024-02-14T17:41:31.025837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:11:24.842191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024079+0000","last_change":"2024-02-14T17:41:04.385500+0000","last_active":"2024-02-14T17:41:31.024079+0000","last_peered":"2024-02-14T17:41:31.024079+0000","last_clean":"2024-02-14T17:41:31.024079+0000","last_became_active":"2024-02-14T17:41:04.384913+0000","last_became_peered":"2024-02-14T17:41:04.384913+0000","last_unstale":"2024-02-14T17:41:31.024079+0000","last_undegraded":"2024-02-14T17:41:31.024079+0000","last_fullsized":"2024-02-14T17:41:31.024079+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:32:47.253287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025188+0000","last_change":"2024-02-14T17:41:08.412099+0000","last_active":"2024-02-14T17:41:31.025188+0000","last_peered":"2024-02-14T17:41:31.025188+0000","last_clean":"2024-02-14T17:41:31.025188+0000","last_became_active":"2024-02-14T17:41:08.411401+0000","last_became_peered":"2024-02-14T17:41:08.411401+0000","last_unstale":"2024-02-14T17:41:31.025188+0000","last_undegraded":"2024-02-14T17:41:31.025188+0000","last_fullsized":"2024-02-14T17:41:31.025188+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:38:20.189799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:42.426416+0000","last_change":"2024-02-14T17:41:06.400291+0000","last_active":"2024-02-14T17:41:42.426416+0000","last_peered":"2024-02-14T17:41:42.426416+0000","last_clean":"2024-02-14T17:41:42.426416+0000","last_became_active":"2024-02-14T17:41:06.399919+0000","last_became_peered":"2024-02-14T17:41:06.399919+0000","last_unstale":"2024-02-14T17:41:42.426416+0000","last_undegraded":"2024-02-14T17:41:42.426416+0000","last_fullsized":"2024-02-14T17:41:42.426416+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:41:06.807684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025247+0000","last_change":"2024-02-14T17:41:02.375597+0000","last_active":"2024-02-14T17:41:31.025247+0000","last_peered":"2024-02-14T17:41:31.025247+0000","last_clean":"2024-02-14T17:41:31.025247+0000","last_became_active":"2024-02-14T17:41:02.374050+0000","last_became_peered":"2024-02-14T17:41:02.374050+0000","last_unstale":"2024-02-14T17:41:31.025247+0000","last_undegraded":"2024-02-14T17:41:31.025247+0000","last_fullsized":"2024-02-14T17:41:31.025247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:14:46.169240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024859+0000","last_change":"2024-02-14T17:41:04.384674+0000","last_active":"2024-02-14T17:41:31.024859+0000","last_peered":"2024-02-14T17:41:31.024859+0000","last_clean":"2024-02-14T17:41:31.024859+0000","last_became_active":"2024-02-14T17:41:04.384273+0000","last_became_peered":"2024-02-14T17:41:04.384273+0000","last_unstale":"2024-02-14T17:41:31.024859+0000","last_undegraded":"2024-02-14T17:41:31.024859+0000","last_fullsized":"2024-02-14T17:41:31.024859+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:59:06.854382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024614+0000","last_change":"2024-02-14T17:41:08.412707+0000","last_active":"2024-02-14T17:41:31.024614+0000","last_peered":"2024-02-14T17:41:31.024614+0000","last_clean":"2024-02-14T17:41:31.024614+0000","last_became_active":"2024-02-14T17:41:08.412329+0000","last_became_peered":"2024-02-14T17:41:08.412329+0000","last_unstale":"2024-02-14T17:41:31.024614+0000","last_undegraded":"2024-02-14T17:41:31.024614+0000","last_fullsized":"2024-02-14T17:41:31.024614+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:55:04.996507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026281+0000","last_change":"2024-02-14T17:41:06.400816+0000","last_active":"2024-02-14T17:41:31.026281+0000","last_peered":"2024-02-14T17:41:31.026281+0000","last_clean":"2024-02-14T17:41:31.026281+0000","last_became_active":"2024-02-14T17:41:06.399580+0000","last_became_peered":"2024-02-14T17:41:06.399580+0000","last_unstale":"2024-02-14T17:41:31.026281+0000","last_undegraded":"2024-02-14T17:41:31.026281+0000","last_fullsized":"2024-02-14T17:41:31.026281+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:13:32.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023804+0000","last_change":"2024-02-14T17:41:02.375123+0000","last_active":"2024-02-14T17:41:31.023804+0000","last_peered":"2024-02-14T17:41:31.023804+0000","last_clean":"2024-02-14T17:41:31.023804+0000","last_became_active":"2024-02-14T17:41:02.374470+0000","last_became_peered":"2024-02-14T17:41:02.374470+0000","last_unstale":"2024-02-14T17:41:31.023804+0000","last_undegraded":"2024-02-14T17:41:31.023804+0000","last_fullsized":"2024-02-14T17:41:31.023804+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:48:56.359866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023034+0000","last_change":"2024-02-14T17:41:04.385521+0000","last_active":"2024-02-14T17:41:31.023034+0000","last_peered":"2024-02-14T17:41:31.023034+0000","last_clean":"2024-02-14T17:41:31.023034+0000","last_became_active":"2024-02-14T17:41:04.384711+0000","last_became_peered":"2024-02-14T17:41:04.384711+0000","last_unstale":"2024-02-14T17:41:31.023034+0000","last_undegraded":"2024-02-14T17:41:31.023034+0000","last_fullsized":"2024-02-14T17:41:31.023034+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:28:46.379419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026478+0000","last_change":"2024-02-14T17:41:08.411168+0000","last_active":"2024-02-14T17:41:31.026478+0000","last_peered":"2024-02-14T17:41:31.026478+0000","last_clean":"2024-02-14T17:41:31.026478+0000","last_became_active":"2024-02-14T17:41:08.410861+0000","last_became_peered":"2024-02-14T17:41:08.410861+0000","last_unstale":"2024-02-14T17:41:31.026478+0000","last_undegraded":"2024-02-14T17:41:31.026478+0000","last_fullsized":"2024-02-14T17:41:31.026478+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:42:56.373255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024247+0000","last_change":"2024-02-14T17:41:06.399345+0000","last_active":"2024-02-14T17:41:31.024247+0000","last_peered":"2024-02-14T17:41:31.024247+0000","last_clean":"2024-02-14T17:41:31.024247+0000","last_became_active":"2024-02-14T17:41:06.398671+0000","last_became_peered":"2024-02-14T17:41:06.398671+0000","last_unstale":"2024-02-14T17:41:31.024247+0000","last_undegraded":"2024-02-14T17:41:31.024247+0000","last_fullsized":"2024-02-14T17:41:31.024247+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:27:35.100179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024838+0000","last_change":"2024-02-14T17:41:02.379654+0000","last_active":"2024-02-14T17:41:31.024838+0000","last_peered":"2024-02-14T17:41:31.024838+0000","last_clean":"2024-02-14T17:41:31.024838+0000","last_became_active":"2024-02-14T17:41:02.379195+0000","last_became_peered":"2024-02-14T17:41:02.379195+0000","last_unstale":"2024-02-14T17:41:31.024838+0000","last_undegraded":"2024-02-14T17:41:31.024838+0000","last_fullsized":"2024-02-14T17:41:31.024838+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:08:14.065622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026238+0000","last_change":"2024-02-14T17:41:04.389438+0000","last_active":"2024-02-14T17:41:31.026238+0000","last_peered":"2024-02-14T17:41:31.026238+0000","last_clean":"2024-02-14T17:41:31.026238+0000","last_became_active":"2024-02-14T17:41:04.389160+0000","last_became_peered":"2024-02-14T17:41:04.389160+0000","last_unstale":"2024-02-14T17:41:31.026238+0000","last_undegraded":"2024-02-14T17:41:31.026238+0000","last_fullsized":"2024-02-14T17:41:31.026238+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:46:55.647026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025940+0000","last_change":"2024-02-14T17:41:08.414494+0000","last_active":"2024-02-14T17:41:31.025940+0000","last_peered":"2024-02-14T17:41:31.025940+0000","last_clean":"2024-02-14T17:41:31.025940+0000","last_became_active":"2024-02-14T17:41:08.414146+0000","last_became_peered":"2024-02-14T17:41:08.414146+0000","last_unstale":"2024-02-14T17:41:31.025940+0000","last_undegraded":"2024-02-14T17:41:31.025940+0000","last_fullsized":"2024-02-14T17:41:31.025940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T17:51:52.672695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026756+0000","last_change":"2024-02-14T17:41:02.379126+0000","last_active":"2024-02-14T17:41:31.026756+0000","last_peered":"2024-02-14T17:41:31.026756+0000","last_clean":"2024-02-14T17:41:31.026756+0000","last_became_active":"2024-02-14T17:41:02.374828+0000","last_became_peered":"2024-02-14T17:41:02.374828+0000","last_unstale":"2024-02-14T17:41:31.026756+0000","last_undegraded":"2024-02-14T17:41:31.026756+0000","last_fullsized":"2024-02-14T17:41:31.026756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:48:58.272502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026198+0000","last_change":"2024-02-14T17:41:04.391336+0000","last_active":"2024-02-14T17:41:31.026198+0000","last_peered":"2024-02-14T17:41:31.026198+0000","last_clean":"2024-02-14T17:41:31.026198+0000","last_became_active":"2024-02-14T17:41:04.390694+0000","last_became_peered":"2024-02-14T17:41:04.390694+0000","last_unstale":"2024-02-14T17:41:31.026198+0000","last_undegraded":"2024-02-14T17:41:31.026198+0000","last_fullsized":"2024-02-14T17:41:31.026198+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:55:21.394060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:42.426458+0000","last_change":"2024-02-14T17:41:06.398177+0000","last_active":"2024-02-14T17:41:42.426458+0000","last_peered":"2024-02-14T17:41:42.426458+0000","last_clean":"2024-02-14T17:41:42.426458+0000","last_became_active":"2024-02-14T17:41:06.397559+0000","last_became_peered":"2024-02-14T17:41:06.397559+0000","last_unstale":"2024-02-14T17:41:42.426458+0000","last_undegraded":"2024-02-14T17:41:42.426458+0000","last_fullsized":"2024-02-14T17:41:42.426458+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:13:13.900330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024910+0000","last_change":"2024-02-14T17:41:08.407061+0000","last_active":"2024-02-14T17:41:31.024910+0000","last_peered":"2024-02-14T17:41:31.024910+0000","last_clean":"2024-02-14T17:41:31.024910+0000","last_became_active":"2024-02-14T17:41:08.406666+0000","last_became_peered":"2024-02-14T17:41:08.406666+0000","last_unstale":"2024-02-14T17:41:31.024910+0000","last_undegraded":"2024-02-14T17:41:31.024910+0000","last_fullsized":"2024-02-14T17:41:31.024910+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:50:42.156106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026367+0000","last_change":"2024-02-14T17:41:02.376978+0000","last_active":"2024-02-14T17:41:31.026367+0000","last_peered":"2024-02-14T17:41:31.026367+0000","last_clean":"2024-02-14T17:41:31.026367+0000","last_became_active":"2024-02-14T17:41:02.376670+0000","last_became_peered":"2024-02-14T17:41:02.376670+0000","last_unstale":"2024-02-14T17:41:31.026367+0000","last_undegraded":"2024-02-14T17:41:31.026367+0000","last_fullsized":"2024-02-14T17:41:31.026367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:27:33.525258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:42.426062+0000","last_change":"2024-02-14T17:41:04.385903+0000","last_active":"2024-02-14T17:41:42.426062+0000","last_peered":"2024-02-14T17:41:42.426062+0000","last_clean":"2024-02-14T17:41:42.426062+0000","last_became_active":"2024-02-14T17:41:04.384273+0000","last_became_peered":"2024-02-14T17:41:04.384273+0000","last_unstale":"2024-02-14T17:41:42.426062+0000","last_undegraded":"2024-02-14T17:41:42.426062+0000","last_fullsized":"2024-02-14T17:41:42.426062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:21:21.135733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025141+0000","last_change":"2024-02-14T17:41:06.397116+0000","last_active":"2024-02-14T17:41:31.025141+0000","last_peered":"2024-02-14T17:41:31.025141+0000","last_clean":"2024-02-14T17:41:31.025141+0000","last_became_active":"2024-02-14T17:41:06.396786+0000","last_became_peered":"2024-02-14T17:41:06.396786+0000","last_unstale":"2024-02-14T17:41:31.025141+0000","last_undegraded":"2024-02-14T17:41:31.025141+0000","last_fullsized":"2024-02-14T17:41:31.025141+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:05:19.292189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025051+0000","last_change":"2024-02-14T17:41:08.410968+0000","last_active":"2024-02-14T17:41:31.025051+0000","last_peered":"2024-02-14T17:41:31.025051+0000","last_clean":"2024-02-14T17:41:31.025051+0000","last_became_active":"2024-02-14T17:41:08.410678+0000","last_became_peered":"2024-02-14T17:41:08.410678+0000","last_unstale":"2024-02-14T17:41:31.025051+0000","last_undegraded":"2024-02-14T17:41:31.025051+0000","last_fullsized":"2024-02-14T17:41:31.025051+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:44:36.538499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024219+0000","last_change":"2024-02-14T17:41:02.375584+0000","last_active":"2024-02-14T17:41:31.024219+0000","last_peered":"2024-02-14T17:41:31.024219+0000","last_clean":"2024-02-14T17:41:31.024219+0000","last_became_active":"2024-02-14T17:41:02.374702+0000","last_became_peered":"2024-02-14T17:41:02.374702+0000","last_unstale":"2024-02-14T17:41:31.024219+0000","last_undegraded":"2024-02-14T17:41:31.024219+0000","last_fullsized":"2024-02-14T17:41:31.024219+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:48:59.956150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024988+0000","last_change":"2024-02-14T17:41:04.384796+0000","last_active":"2024-02-14T17:41:31.024988+0000","last_peered":"2024-02-14T17:41:31.024988+0000","last_clean":"2024-02-14T17:41:31.024988+0000","last_became_active":"2024-02-14T17:41:04.384460+0000","last_became_peered":"2024-02-14T17:41:04.384460+0000","last_unstale":"2024-02-14T17:41:31.024988+0000","last_undegraded":"2024-02-14T17:41:31.024988+0000","last_fullsized":"2024-02-14T17:41:31.024988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:21:22.165443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025678+0000","last_change":"2024-02-14T17:41:06.399414+0000","last_active":"2024-02-14T17:41:31.025678+0000","last_peered":"2024-02-14T17:41:31.025678+0000","last_clean":"2024-02-14T17:41:31.025678+0000","last_became_active":"2024-02-14T17:41:06.398893+0000","last_became_peered":"2024-02-14T17:41:06.398893+0000","last_unstale":"2024-02-14T17:41:31.025678+0000","last_undegraded":"2024-02-14T17:41:31.025678+0000","last_fullsized":"2024-02-14T17:41:31.025678+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:32:18.973956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025605+0000","last_change":"2024-02-14T17:41:08.412220+0000","last_active":"2024-02-14T17:41:31.025605+0000","last_peered":"2024-02-14T17:41:31.025605+0000","last_clean":"2024-02-14T17:41:31.025605+0000","last_became_active":"2024-02-14T17:41:08.411861+0000","last_became_peered":"2024-02-14T17:41:08.411861+0000","last_unstale":"2024-02-14T17:41:31.025605+0000","last_undegraded":"2024-02-14T17:41:31.025605+0000","last_fullsized":"2024-02-14T17:41:31.025605+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:50:49.759434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024971+0000","last_change":"2024-02-14T17:41:04.391691+0000","last_active":"2024-02-14T17:41:31.024971+0000","last_peered":"2024-02-14T17:41:31.024971+0000","last_clean":"2024-02-14T17:41:31.024971+0000","last_became_active":"2024-02-14T17:41:04.390481+0000","last_became_peered":"2024-02-14T17:41:04.390481+0000","last_unstale":"2024-02-14T17:41:31.024971+0000","last_undegraded":"2024-02-14T17:41:31.024971+0000","last_fullsized":"2024-02-14T17:41:31.024971+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:07:37.272279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023608+0000","last_change":"2024-02-14T17:41:02.378514+0000","last_active":"2024-02-14T17:41:31.023608+0000","last_peered":"2024-02-14T17:41:31.023608+0000","last_clean":"2024-02-14T17:41:31.023608+0000","last_became_active":"2024-02-14T17:41:02.378164+0000","last_became_peered":"2024-02-14T17:41:02.378164+0000","last_unstale":"2024-02-14T17:41:31.023608+0000","last_undegraded":"2024-02-14T17:41:31.023608+0000","last_fullsized":"2024-02-14T17:41:31.023608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:32:02.997296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027217+0000","last_change":"2024-02-14T17:41:06.402091+0000","last_active":"2024-02-14T17:41:31.027217+0000","last_peered":"2024-02-14T17:41:31.027217+0000","last_clean":"2024-02-14T17:41:31.027217+0000","last_became_active":"2024-02-14T17:41:06.401691+0000","last_became_peered":"2024-02-14T17:41:06.401691+0000","last_unstale":"2024-02-14T17:41:31.027217+0000","last_undegraded":"2024-02-14T17:41:31.027217+0000","last_fullsized":"2024-02-14T17:41:31.027217+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T05:39:34.932877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025437+0000","last_change":"2024-02-14T17:41:08.412755+0000","last_active":"2024-02-14T17:41:31.025437+0000","last_peered":"2024-02-14T17:41:31.025437+0000","last_clean":"2024-02-14T17:41:31.025437+0000","last_became_active":"2024-02-14T17:41:08.412244+0000","last_became_peered":"2024-02-14T17:41:08.412244+0000","last_unstale":"2024-02-14T17:41:31.025437+0000","last_undegraded":"2024-02-14T17:41:31.025437+0000","last_fullsized":"2024-02-14T17:41:31.025437+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:01:12.104068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023881+0000","last_change":"2024-02-14T17:41:04.386897+0000","last_active":"2024-02-14T17:41:31.023881+0000","last_peered":"2024-02-14T17:41:31.023881+0000","last_clean":"2024-02-14T17:41:31.023881+0000","last_became_active":"2024-02-14T17:41:04.386157+0000","last_became_peered":"2024-02-14T17:41:04.386157+0000","last_unstale":"2024-02-14T17:41:31.023881+0000","last_undegraded":"2024-02-14T17:41:31.023881+0000","last_fullsized":"2024-02-14T17:41:31.023881+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:01:29.518979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024698+0000","last_change":"2024-02-14T17:41:02.376322+0000","last_active":"2024-02-14T17:41:31.024698+0000","last_peered":"2024-02-14T17:41:31.024698+0000","last_clean":"2024-02-14T17:41:31.024698+0000","last_became_active":"2024-02-14T17:41:02.375044+0000","last_became_peered":"2024-02-14T17:41:02.375044+0000","last_unstale":"2024-02-14T17:41:31.024698+0000","last_undegraded":"2024-02-14T17:41:31.024698+0000","last_fullsized":"2024-02-14T17:41:31.024698+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:02:55.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:33.127633+0000","last_change":"2024-02-14T17:40:49.993958+0000","last_active":"2024-02-14T17:41:33.127633+0000","last_peered":"2024-02-14T17:41:33.127633+0000","last_clean":"2024-02-14T17:41:33.127633+0000","last_became_active":"2024-02-14T17:40:49.993641+0000","last_became_peered":"2024-02-14T17:40:49.993641+0000","last_unstale":"2024-02-14T17:41:33.127633+0000","last_undegraded":"2024-02-14T17:41:33.127633+0000","last_fullsized":"2024-02-14T17:41:33.127633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:40:48.979634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:40:48.979634+0000","last_clean_scrub_stamp":"2024-02-14T17:40:48.979634+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:27:32.829382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025947+0000","last_change":"2024-02-14T17:41:06.401781+0000","last_active":"2024-02-14T17:41:31.025947+0000","last_peered":"2024-02-14T17:41:31.025947+0000","last_clean":"2024-02-14T17:41:31.025947+0000","last_became_active":"2024-02-14T17:41:06.401458+0000","last_became_peered":"2024-02-14T17:41:06.401458+0000","last_unstale":"2024-02-14T17:41:31.025947+0000","last_undegraded":"2024-02-14T17:41:31.025947+0000","last_fullsized":"2024-02-14T17:41:31.025947+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:56:20.668201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024361+0000","last_change":"2024-02-14T17:41:08.411284+0000","last_active":"2024-02-14T17:41:31.024361+0000","last_peered":"2024-02-14T17:41:31.024361+0000","last_clean":"2024-02-14T17:41:31.024361+0000","last_became_active":"2024-02-14T17:41:08.409981+0000","last_became_peered":"2024-02-14T17:41:08.409981+0000","last_unstale":"2024-02-14T17:41:31.024361+0000","last_undegraded":"2024-02-14T17:41:31.024361+0000","last_fullsized":"2024-02-14T17:41:31.024361+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:57:18.029925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025054+0000","last_change":"2024-02-14T17:41:04.385719+0000","last_active":"2024-02-14T17:41:31.025054+0000","last_peered":"2024-02-14T17:41:31.025054+0000","last_clean":"2024-02-14T17:41:31.025054+0000","last_became_active":"2024-02-14T17:41:04.385254+0000","last_became_peered":"2024-02-14T17:41:04.385254+0000","last_unstale":"2024-02-14T17:41:31.025054+0000","last_undegraded":"2024-02-14T17:41:31.025054+0000","last_fullsized":"2024-02-14T17:41:31.025054+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:16:28.148386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023062+0000","last_change":"2024-02-14T17:41:02.376882+0000","last_active":"2024-02-14T17:41:31.023062+0000","last_peered":"2024-02-14T17:41:31.023062+0000","last_clean":"2024-02-14T17:41:31.023062+0000","last_became_active":"2024-02-14T17:41:02.375178+0000","last_became_peered":"2024-02-14T17:41:02.375178+0000","last_unstale":"2024-02-14T17:41:31.023062+0000","last_undegraded":"2024-02-14T17:41:31.023062+0000","last_fullsized":"2024-02-14T17:41:31.023062+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:27:45.838111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025977+0000","last_change":"2024-02-14T17:41:06.398222+0000","last_active":"2024-02-14T17:41:31.025977+0000","last_peered":"2024-02-14T17:41:31.025977+0000","last_clean":"2024-02-14T17:41:31.025977+0000","last_became_active":"2024-02-14T17:41:06.397673+0000","last_became_peered":"2024-02-14T17:41:06.397673+0000","last_unstale":"2024-02-14T17:41:31.025977+0000","last_undegraded":"2024-02-14T17:41:31.025977+0000","last_fullsized":"2024-02-14T17:41:31.025977+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:01:28.963313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023939+0000","last_change":"2024-02-14T17:41:08.408359+0000","last_active":"2024-02-14T17:41:31.023939+0000","last_peered":"2024-02-14T17:41:31.023939+0000","last_clean":"2024-02-14T17:41:31.023939+0000","last_became_active":"2024-02-14T17:41:08.407915+0000","last_became_peered":"2024-02-14T17:41:08.407915+0000","last_unstale":"2024-02-14T17:41:31.023939+0000","last_undegraded":"2024-02-14T17:41:31.023939+0000","last_fullsized":"2024-02-14T17:41:31.023939+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:42:45.553065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027783+0000","last_change":"2024-02-14T17:41:02.379121+0000","last_active":"2024-02-14T17:41:31.027783+0000","last_peered":"2024-02-14T17:41:31.027783+0000","last_clean":"2024-02-14T17:41:31.027783+0000","last_became_active":"2024-02-14T17:41:02.377718+0000","last_became_peered":"2024-02-14T17:41:02.377718+0000","last_unstale":"2024-02-14T17:41:31.027783+0000","last_undegraded":"2024-02-14T17:41:31.027783+0000","last_fullsized":"2024-02-14T17:41:31.027783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:45:49.190070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:42.426079+0000","last_change":"2024-02-14T17:41:04.387001+0000","last_active":"2024-02-14T17:41:42.426079+0000","last_peered":"2024-02-14T17:41:42.426079+0000","last_clean":"2024-02-14T17:41:42.426079+0000","last_became_active":"2024-02-14T17:41:04.386247+0000","last_became_peered":"2024-02-14T17:41:04.386247+0000","last_unstale":"2024-02-14T17:41:42.426079+0000","last_undegraded":"2024-02-14T17:41:42.426079+0000","last_fullsized":"2024-02-14T17:41:42.426079+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:12:23.487876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:42.426429+0000","last_change":"2024-02-14T17:41:06.397799+0000","last_active":"2024-02-14T17:41:42.426429+0000","last_peered":"2024-02-14T17:41:42.426429+0000","last_clean":"2024-02-14T17:41:42.426429+0000","last_became_active":"2024-02-14T17:41:06.397505+0000","last_became_peered":"2024-02-14T17:41:06.397505+0000","last_unstale":"2024-02-14T17:41:42.426429+0000","last_undegraded":"2024-02-14T17:41:42.426429+0000","last_fullsized":"2024-02-14T17:41:42.426429+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:46:53.143729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023142+0000","last_change":"2024-02-14T17:41:08.410153+0000","last_active":"2024-02-14T17:41:31.023142+0000","last_peered":"2024-02-14T17:41:31.023142+0000","last_clean":"2024-02-14T17:41:31.023142+0000","last_became_active":"2024-02-14T17:41:08.409629+0000","last_became_peered":"2024-02-14T17:41:08.409629+0000","last_unstale":"2024-02-14T17:41:31.023142+0000","last_undegraded":"2024-02-14T17:41:31.023142+0000","last_fullsized":"2024-02-14T17:41:31.023142+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T17:46:34.678202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025497+0000","last_change":"2024-02-14T17:41:02.378465+0000","last_active":"2024-02-14T17:41:31.025497+0000","last_peered":"2024-02-14T17:41:31.025497+0000","last_clean":"2024-02-14T17:41:31.025497+0000","last_became_active":"2024-02-14T17:41:02.378118+0000","last_became_peered":"2024-02-14T17:41:02.378118+0000","last_unstale":"2024-02-14T17:41:31.025497+0000","last_undegraded":"2024-02-14T17:41:31.025497+0000","last_fullsized":"2024-02-14T17:41:31.025497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:07:09.904389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027470+0000","last_change":"2024-02-14T17:41:04.389601+0000","last_active":"2024-02-14T17:41:31.027470+0000","last_peered":"2024-02-14T17:41:31.027470+0000","last_clean":"2024-02-14T17:41:31.027470+0000","last_became_active":"2024-02-14T17:41:04.389000+0000","last_became_peered":"2024-02-14T17:41:04.389000+0000","last_unstale":"2024-02-14T17:41:31.027470+0000","last_undegraded":"2024-02-14T17:41:31.027470+0000","last_fullsized":"2024-02-14T17:41:31.027470+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:21:05.973368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:42.425982+0000","last_change":"2024-02-14T17:41:06.398490+0000","last_active":"2024-02-14T17:41:42.425982+0000","last_peered":"2024-02-14T17:41:42.425982+0000","last_clean":"2024-02-14T17:41:42.425982+0000","last_became_active":"2024-02-14T17:41:06.397851+0000","last_became_peered":"2024-02-14T17:41:06.397851+0000","last_unstale":"2024-02-14T17:41:42.425982+0000","last_undegraded":"2024-02-14T17:41:42.425982+0000","last_fullsized":"2024-02-14T17:41:42.425982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:58:32.448296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025848+0000","last_change":"2024-02-14T17:41:08.413803+0000","last_active":"2024-02-14T17:41:31.025848+0000","last_peered":"2024-02-14T17:41:31.025848+0000","last_clean":"2024-02-14T17:41:31.025848+0000","last_became_active":"2024-02-14T17:41:08.413410+0000","last_became_peered":"2024-02-14T17:41:08.413410+0000","last_unstale":"2024-02-14T17:41:31.025848+0000","last_undegraded":"2024-02-14T17:41:31.025848+0000","last_fullsized":"2024-02-14T17:41:31.025848+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:01:10.293991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024655+0000","last_change":"2024-02-14T17:41:02.376920+0000","last_active":"2024-02-14T17:41:31.024655+0000","last_peered":"2024-02-14T17:41:31.024655+0000","last_clean":"2024-02-14T17:41:31.024655+0000","last_became_active":"2024-02-14T17:41:02.376259+0000","last_became_peered":"2024-02-14T17:41:02.376259+0000","last_unstale":"2024-02-14T17:41:31.024655+0000","last_undegraded":"2024-02-14T17:41:31.024655+0000","last_fullsized":"2024-02-14T17:41:31.024655+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:03:42.033670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025123+0000","last_change":"2024-02-14T17:41:04.385593+0000","last_active":"2024-02-14T17:41:31.025123+0000","last_peered":"2024-02-14T17:41:31.025123+0000","last_clean":"2024-02-14T17:41:31.025123+0000","last_became_active":"2024-02-14T17:41:04.384985+0000","last_became_peered":"2024-02-14T17:41:04.384985+0000","last_unstale":"2024-02-14T17:41:31.025123+0000","last_undegraded":"2024-02-14T17:41:31.025123+0000","last_fullsized":"2024-02-14T17:41:31.025123+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:43:15.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024964+0000","last_change":"2024-02-14T17:41:06.398676+0000","last_active":"2024-02-14T17:41:31.024964+0000","last_peered":"2024-02-14T17:41:31.024964+0000","last_clean":"2024-02-14T17:41:31.024964+0000","last_became_active":"2024-02-14T17:41:06.398446+0000","last_became_peered":"2024-02-14T17:41:06.398446+0000","last_unstale":"2024-02-14T17:41:31.024964+0000","last_undegraded":"2024-02-14T17:41:31.024964+0000","last_fullsized":"2024-02-14T17:41:31.024964+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:23:49.110802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024186+0000","last_change":"2024-02-14T17:41:08.412695+0000","last_active":"2024-02-14T17:41:31.024186+0000","last_peered":"2024-02-14T17:41:31.024186+0000","last_clean":"2024-02-14T17:41:31.024186+0000","last_became_active":"2024-02-14T17:41:08.412307+0000","last_became_peered":"2024-02-14T17:41:08.412307+0000","last_unstale":"2024-02-14T17:41:31.024186+0000","last_undegraded":"2024-02-14T17:41:31.024186+0000","last_fullsized":"2024-02-14T17:41:31.024186+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:07:51.183677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:42.426097+0000","last_change":"2024-02-14T17:41:06.404301+0000","last_active":"2024-02-14T17:41:42.426097+0000","last_peered":"2024-02-14T17:41:42.426097+0000","last_clean":"2024-02-14T17:41:42.426097+0000","last_became_active":"2024-02-14T17:41:06.403971+0000","last_became_peered":"2024-02-14T17:41:06.403971+0000","last_unstale":"2024-02-14T17:41:42.426097+0000","last_undegraded":"2024-02-14T17:41:42.426097+0000","last_fullsized":"2024-02-14T17:41:42.426097+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:53:58.588560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023234+0000","last_change":"2024-02-14T17:41:02.377863+0000","last_active":"2024-02-14T17:41:31.023234+0000","last_peered":"2024-02-14T17:41:31.023234+0000","last_clean":"2024-02-14T17:41:31.023234+0000","last_became_active":"2024-02-14T17:41:02.377557+0000","last_became_peered":"2024-02-14T17:41:02.377557+0000","last_unstale":"2024-02-14T17:41:31.023234+0000","last_undegraded":"2024-02-14T17:41:31.023234+0000","last_fullsized":"2024-02-14T17:41:31.023234+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:13:27.575502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026557+0000","last_change":"2024-02-14T17:41:04.391728+0000","last_active":"2024-02-14T17:41:31.026557+0000","last_peered":"2024-02-14T17:41:31.026557+0000","last_clean":"2024-02-14T17:41:31.026557+0000","last_became_active":"2024-02-14T17:41:04.391238+0000","last_became_peered":"2024-02-14T17:41:04.391238+0000","last_unstale":"2024-02-14T17:41:31.026557+0000","last_undegraded":"2024-02-14T17:41:31.026557+0000","last_fullsized":"2024-02-14T17:41:31.026557+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:34:49.987772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026345+0000","last_change":"2024-02-14T17:41:08.413366+0000","last_active":"2024-02-14T17:41:31.026345+0000","last_peered":"2024-02-14T17:41:31.026345+0000","last_clean":"2024-02-14T17:41:31.026345+0000","last_became_active":"2024-02-14T17:41:08.413097+0000","last_became_peered":"2024-02-14T17:41:08.413097+0000","last_unstale":"2024-02-14T17:41:31.026345+0000","last_undegraded":"2024-02-14T17:41:31.026345+0000","last_fullsized":"2024-02-14T17:41:31.026345+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:09:05.406307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026378+0000","last_change":"2024-02-14T17:41:06.401347+0000","last_active":"2024-02-14T17:41:31.026378+0000","last_peered":"2024-02-14T17:41:31.026378+0000","last_clean":"2024-02-14T17:41:31.026378+0000","last_became_active":"2024-02-14T17:41:06.400975+0000","last_became_peered":"2024-02-14T17:41:06.400975+0000","last_unstale":"2024-02-14T17:41:31.026378+0000","last_undegraded":"2024-02-14T17:41:31.026378+0000","last_fullsized":"2024-02-14T17:41:31.026378+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:05:03.924070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023832+0000","last_change":"2024-02-14T17:41:02.376245+0000","last_active":"2024-02-14T17:41:31.023832+0000","last_peered":"2024-02-14T17:41:31.023832+0000","last_clean":"2024-02-14T17:41:31.023832+0000","last_became_active":"2024-02-14T17:41:02.375782+0000","last_became_peered":"2024-02-14T17:41:02.375782+0000","last_unstale":"2024-02-14T17:41:31.023832+0000","last_undegraded":"2024-02-14T17:41:31.023832+0000","last_fullsized":"2024-02-14T17:41:31.023832+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:06:01.698681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025051+0000","last_change":"2024-02-14T17:41:04.391987+0000","last_active":"2024-02-14T17:41:31.025051+0000","last_peered":"2024-02-14T17:41:31.025051+0000","last_clean":"2024-02-14T17:41:31.025051+0000","last_became_active":"2024-02-14T17:41:04.391459+0000","last_became_peered":"2024-02-14T17:41:04.391459+0000","last_unstale":"2024-02-14T17:41:31.025051+0000","last_undegraded":"2024-02-14T17:41:31.025051+0000","last_fullsized":"2024-02-14T17:41:31.025051+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:08:54.650126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026567+0000","last_change":"2024-02-14T17:41:08.413463+0000","last_active":"2024-02-14T17:41:31.026567+0000","last_peered":"2024-02-14T17:41:31.026567+0000","last_clean":"2024-02-14T17:41:31.026567+0000","last_became_active":"2024-02-14T17:41:08.413167+0000","last_became_peered":"2024-02-14T17:41:08.413167+0000","last_unstale":"2024-02-14T17:41:31.026567+0000","last_undegraded":"2024-02-14T17:41:31.026567+0000","last_fullsized":"2024-02-14T17:41:31.026567+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:15:20.187431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025679+0000","last_change":"2024-02-14T17:41:02.377592+0000","last_active":"2024-02-14T17:41:31.025679+0000","last_peered":"2024-02-14T17:41:31.025679+0000","last_clean":"2024-02-14T17:41:31.025679+0000","last_became_active":"2024-02-14T17:41:02.377236+0000","last_became_peered":"2024-02-14T17:41:02.377236+0000","last_unstale":"2024-02-14T17:41:31.025679+0000","last_undegraded":"2024-02-14T17:41:31.025679+0000","last_fullsized":"2024-02-14T17:41:31.025679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:16:15.336245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024156+0000","last_change":"2024-02-14T17:41:04.384460+0000","last_active":"2024-02-14T17:41:31.024156+0000","last_peered":"2024-02-14T17:41:31.024156+0000","last_clean":"2024-02-14T17:41:31.024156+0000","last_became_active":"2024-02-14T17:41:04.384105+0000","last_became_peered":"2024-02-14T17:41:04.384105+0000","last_unstale":"2024-02-14T17:41:31.024156+0000","last_undegraded":"2024-02-14T17:41:31.024156+0000","last_fullsized":"2024-02-14T17:41:31.024156+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:44:48.276948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025807+0000","last_change":"2024-02-14T17:41:06.401122+0000","last_active":"2024-02-14T17:41:31.025807+0000","last_peered":"2024-02-14T17:41:31.025807+0000","last_clean":"2024-02-14T17:41:31.025807+0000","last_became_active":"2024-02-14T17:41:06.400537+0000","last_became_peered":"2024-02-14T17:41:06.400537+0000","last_unstale":"2024-02-14T17:41:31.025807+0000","last_undegraded":"2024-02-14T17:41:31.025807+0000","last_fullsized":"2024-02-14T17:41:31.025807+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:20:35.611589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023660+0000","last_change":"2024-02-14T17:41:08.409598+0000","last_active":"2024-02-14T17:41:31.023660+0000","last_peered":"2024-02-14T17:41:31.023660+0000","last_clean":"2024-02-14T17:41:31.023660+0000","last_became_active":"2024-02-14T17:41:08.409348+0000","last_became_peered":"2024-02-14T17:41:08.409348+0000","last_unstale":"2024-02-14T17:41:31.023660+0000","last_undegraded":"2024-02-14T17:41:31.023660+0000","last_fullsized":"2024-02-14T17:41:31.023660+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T05:19:02.194386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:42.426054+0000","last_change":"2024-02-14T17:41:06.401108+0000","last_active":"2024-02-14T17:41:42.426054+0000","last_peered":"2024-02-14T17:41:42.426054+0000","last_clean":"2024-02-14T17:41:42.426054+0000","last_became_active":"2024-02-14T17:41:06.400711+0000","last_became_peered":"2024-02-14T17:41:06.400711+0000","last_unstale":"2024-02-14T17:41:42.426054+0000","last_undegraded":"2024-02-14T17:41:42.426054+0000","last_fullsized":"2024-02-14T17:41:42.426054+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:22:11.094830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025639+0000","last_change":"2024-02-14T17:41:02.375129+0000","last_active":"2024-02-14T17:41:31.025639+0000","last_peered":"2024-02-14T17:41:31.025639+0000","last_clean":"2024-02-14T17:41:31.025639+0000","last_became_active":"2024-02-14T17:41:02.374404+0000","last_became_peered":"2024-02-14T17:41:02.374404+0000","last_unstale":"2024-02-14T17:41:31.025639+0000","last_undegraded":"2024-02-14T17:41:31.025639+0000","last_fullsized":"2024-02-14T17:41:31.025639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:28:47.630254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024400+0000","last_change":"2024-02-14T17:41:04.387403+0000","last_active":"2024-02-14T17:41:31.024400+0000","last_peered":"2024-02-14T17:41:31.024400+0000","last_clean":"2024-02-14T17:41:31.024400+0000","last_became_active":"2024-02-14T17:41:04.386110+0000","last_became_peered":"2024-02-14T17:41:04.386110+0000","last_unstale":"2024-02-14T17:41:31.024400+0000","last_undegraded":"2024-02-14T17:41:31.024400+0000","last_fullsized":"2024-02-14T17:41:31.024400+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:46:24.866488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025354+0000","last_change":"2024-02-14T17:41:08.412661+0000","last_active":"2024-02-14T17:41:31.025354+0000","last_peered":"2024-02-14T17:41:31.025354+0000","last_clean":"2024-02-14T17:41:31.025354+0000","last_became_active":"2024-02-14T17:41:08.412353+0000","last_became_peered":"2024-02-14T17:41:08.412353+0000","last_unstale":"2024-02-14T17:41:31.025354+0000","last_undegraded":"2024-02-14T17:41:31.025354+0000","last_fullsized":"2024-02-14T17:41:31.025354+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:04:45.200992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027376+0000","last_change":"2024-02-14T17:41:02.374480+0000","last_active":"2024-02-14T17:41:31.027376+0000","last_peered":"2024-02-14T17:41:31.027376+0000","last_clean":"2024-02-14T17:41:31.027376+0000","last_became_active":"2024-02-14T17:41:02.374194+0000","last_became_peered":"2024-02-14T17:41:02.374194+0000","last_unstale":"2024-02-14T17:41:31.027376+0000","last_undegraded":"2024-02-14T17:41:31.027376+0000","last_fullsized":"2024-02-14T17:41:31.027376+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:05:50.393961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026158+0000","last_change":"2024-02-14T17:41:04.388822+0000","last_active":"2024-02-14T17:41:31.026158+0000","last_peered":"2024-02-14T17:41:31.026158+0000","last_clean":"2024-02-14T17:41:31.026158+0000","last_became_active":"2024-02-14T17:41:04.386893+0000","last_became_peered":"2024-02-14T17:41:04.386893+0000","last_unstale":"2024-02-14T17:41:31.026158+0000","last_undegraded":"2024-02-14T17:41:31.026158+0000","last_fullsized":"2024-02-14T17:41:31.026158+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:28:55.892837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023770+0000","last_change":"2024-02-14T17:41:06.402027+0000","last_active":"2024-02-14T17:41:31.023770+0000","last_peered":"2024-02-14T17:41:31.023770+0000","last_clean":"2024-02-14T17:41:31.023770+0000","last_became_active":"2024-02-14T17:41:06.401623+0000","last_became_peered":"2024-02-14T17:41:06.401623+0000","last_unstale":"2024-02-14T17:41:31.023770+0000","last_undegraded":"2024-02-14T17:41:31.023770+0000","last_fullsized":"2024-02-14T17:41:31.023770+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:15:21.861943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023231+0000","last_change":"2024-02-14T17:41:08.407146+0000","last_active":"2024-02-14T17:41:31.023231+0000","last_peered":"2024-02-14T17:41:31.023231+0000","last_clean":"2024-02-14T17:41:31.023231+0000","last_became_active":"2024-02-14T17:41:08.406753+0000","last_became_peered":"2024-02-14T17:41:08.406753+0000","last_unstale":"2024-02-14T17:41:31.023231+0000","last_undegraded":"2024-02-14T17:41:31.023231+0000","last_fullsized":"2024-02-14T17:41:31.023231+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:17:12.827760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027691+0000","last_change":"2024-02-14T17:41:02.379459+0000","last_active":"2024-02-14T17:41:31.027691+0000","last_peered":"2024-02-14T17:41:31.027691+0000","last_clean":"2024-02-14T17:41:31.027691+0000","last_became_active":"2024-02-14T17:41:02.379144+0000","last_became_peered":"2024-02-14T17:41:02.379144+0000","last_unstale":"2024-02-14T17:41:31.027691+0000","last_undegraded":"2024-02-14T17:41:31.027691+0000","last_fullsized":"2024-02-14T17:41:31.027691+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:51:37.479914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025999+0000","last_change":"2024-02-14T17:41:04.389949+0000","last_active":"2024-02-14T17:41:31.025999+0000","last_peered":"2024-02-14T17:41:31.025999+0000","last_clean":"2024-02-14T17:41:31.025999+0000","last_became_active":"2024-02-14T17:41:04.389386+0000","last_became_peered":"2024-02-14T17:41:04.389386+0000","last_unstale":"2024-02-14T17:41:31.025999+0000","last_undegraded":"2024-02-14T17:41:31.025999+0000","last_fullsized":"2024-02-14T17:41:31.025999+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:54:29.615217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023802+0000","last_change":"2024-02-14T17:41:06.399367+0000","last_active":"2024-02-14T17:41:31.023802+0000","last_peered":"2024-02-14T17:41:31.023802+0000","last_clean":"2024-02-14T17:41:31.023802+0000","last_became_active":"2024-02-14T17:41:06.398644+0000","last_became_peered":"2024-02-14T17:41:06.398644+0000","last_unstale":"2024-02-14T17:41:31.023802+0000","last_undegraded":"2024-02-14T17:41:31.023802+0000","last_fullsized":"2024-02-14T17:41:31.023802+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:27:01.293159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.022983+0000","last_change":"2024-02-14T17:41:08.410044+0000","last_active":"2024-02-14T17:41:31.022983+0000","last_peered":"2024-02-14T17:41:31.022983+0000","last_clean":"2024-02-14T17:41:31.022983+0000","last_became_active":"2024-02-14T17:41:08.409521+0000","last_became_peered":"2024-02-14T17:41:08.409521+0000","last_unstale":"2024-02-14T17:41:31.022983+0000","last_undegraded":"2024-02-14T17:41:31.022983+0000","last_fullsized":"2024-02-14T17:41:31.022983+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:40:20.606034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025024+0000","last_change":"2024-02-14T17:41:02.376503+0000","last_active":"2024-02-14T17:41:31.025024+0000","last_peered":"2024-02-14T17:41:31.025024+0000","last_clean":"2024-02-14T17:41:31.025024+0000","last_became_active":"2024-02-14T17:41:02.376195+0000","last_became_peered":"2024-02-14T17:41:02.376195+0000","last_unstale":"2024-02-14T17:41:31.025024+0000","last_undegraded":"2024-02-14T17:41:31.025024+0000","last_fullsized":"2024-02-14T17:41:31.025024+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:04:35.248705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024894+0000","last_change":"2024-02-14T17:41:04.384945+0000","last_active":"2024-02-14T17:41:31.024894+0000","last_peered":"2024-02-14T17:41:31.024894+0000","last_clean":"2024-02-14T17:41:31.024894+0000","last_became_active":"2024-02-14T17:41:04.384213+0000","last_became_peered":"2024-02-14T17:41:04.384213+0000","last_unstale":"2024-02-14T17:41:31.024894+0000","last_undegraded":"2024-02-14T17:41:31.024894+0000","last_fullsized":"2024-02-14T17:41:31.024894+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:53:48.390856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025290+0000","last_change":"2024-02-14T17:41:06.398471+0000","last_active":"2024-02-14T17:41:31.025290+0000","last_peered":"2024-02-14T17:41:31.025290+0000","last_clean":"2024-02-14T17:41:31.025290+0000","last_became_active":"2024-02-14T17:41:06.397810+0000","last_became_peered":"2024-02-14T17:41:06.397810+0000","last_unstale":"2024-02-14T17:41:31.025290+0000","last_undegraded":"2024-02-14T17:41:31.025290+0000","last_fullsized":"2024-02-14T17:41:31.025290+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:36:46.024003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411343,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed Feb 14 17:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.432}]}]},{"osd":0,"up_from":8,"seq":34359738394,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 14 17:41:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Wed Feb 14 17:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.441},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.49099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.29999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-14T17:41:55.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-14T17:41:56.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-mon[25828]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:56.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-mon[25828]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-14T17:41:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:41:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-mon[25828]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:41:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-mon[25828]: Deploying daemon grafana.a on smithi040 2024-02-14T17:41:56.236 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:41:56.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:56 smithi040 ceph-mon[27571]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:56.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:56 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:56 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:56 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:56 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:56 smithi040 ceph-mon[27571]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-14T17:41:56.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:56 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:56 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:56 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:41:56.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:56 smithi040 ceph-mon[27571]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:41:56.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:56 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:56 smithi040 ceph-mon[27571]: Deploying daemon grafana.a on smithi040 2024-02-14T17:41:56.495 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: ::ffff:172.21.15.40 - - [14/Feb/2024:17:41:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-14T17:41:56.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:56 smithi096 ceph-mon[27796]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:56.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:56 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:56 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:56 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:56 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:56 smithi096 ceph-mon[27796]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-14T17:41:56.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:56 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:56 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:56 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:41:56.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:56 smithi096 ceph-mon[27796]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:41:56.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:56 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:56.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:56 smithi096 ceph-mon[27796]: Deploying daemon grafana.a on smithi040 2024-02-14T17:41:57.119 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:41:56 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[48745]: ts=2024-02-14T17:41:56.969Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000215845s 2024-02-14T17:41:57.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:57 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:57.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:57 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:57.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:57 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:58.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:58 smithi040 ceph-mon[27571]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:58.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:41:58 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:58.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:58 smithi027 ceph-mon[25828]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:58.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:41:58 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:41:58.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:58 smithi096 ceph-mon[27796]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:41:58.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:41:58 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:42:00.118 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-02-14T17:42:00.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:00 smithi027 ceph-mon[25828]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:00.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:00 smithi040 ceph-mon[27571]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:00.484 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:00 smithi096 ceph-mon[27796]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:01.055 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-02-14T17:41:59.041029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96620,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126548,"statfs":{"total":287972524032,"available":287873585152,"internally_reserved":0,"allocated":3158016,"data_stored":2100345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003284"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023532+0000","last_change":"2024-02-14T17:41:08.409572+0000","last_active":"2024-02-14T17:41:31.023532+0000","last_peered":"2024-02-14T17:41:31.023532+0000","last_clean":"2024-02-14T17:41:31.023532+0000","last_became_active":"2024-02-14T17:41:08.409197+0000","last_became_peered":"2024-02-14T17:41:08.409197+0000","last_unstale":"2024-02-14T17:41:31.023532+0000","last_undegraded":"2024-02-14T17:41:31.023532+0000","last_fullsized":"2024-02-14T17:41:31.023532+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:52:18.272045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024775+0000","last_change":"2024-02-14T17:41:06.398484+0000","last_active":"2024-02-14T17:41:31.024775+0000","last_peered":"2024-02-14T17:41:31.024775+0000","last_clean":"2024-02-14T17:41:31.024775+0000","last_became_active":"2024-02-14T17:41:06.397773+0000","last_became_peered":"2024-02-14T17:41:06.397773+0000","last_unstale":"2024-02-14T17:41:31.024775+0000","last_undegraded":"2024-02-14T17:41:31.024775+0000","last_fullsized":"2024-02-14T17:41:31.024775+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:44:45.584250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027399+0000","last_change":"2024-02-14T17:41:02.379405+0000","last_active":"2024-02-14T17:41:31.027399+0000","last_peered":"2024-02-14T17:41:31.027399+0000","last_clean":"2024-02-14T17:41:31.027399+0000","last_became_active":"2024-02-14T17:41:02.379088+0000","last_became_peered":"2024-02-14T17:41:02.379088+0000","last_unstale":"2024-02-14T17:41:31.027399+0000","last_undegraded":"2024-02-14T17:41:31.027399+0000","last_fullsized":"2024-02-14T17:41:31.027399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:41:52.534467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024894+0000","last_change":"2024-02-14T17:41:04.384996+0000","last_active":"2024-02-14T17:41:31.024894+0000","last_peered":"2024-02-14T17:41:31.024894+0000","last_clean":"2024-02-14T17:41:31.024894+0000","last_became_active":"2024-02-14T17:41:04.384539+0000","last_became_peered":"2024-02-14T17:41:04.384539+0000","last_unstale":"2024-02-14T17:41:31.024894+0000","last_undegraded":"2024-02-14T17:41:31.024894+0000","last_fullsized":"2024-02-14T17:41:31.024894+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:36:56.773302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023725+0000","last_change":"2024-02-14T17:41:08.408347+0000","last_active":"2024-02-14T17:41:31.023725+0000","last_peered":"2024-02-14T17:41:31.023725+0000","last_clean":"2024-02-14T17:41:31.023725+0000","last_became_active":"2024-02-14T17:41:08.407865+0000","last_became_peered":"2024-02-14T17:41:08.407865+0000","last_unstale":"2024-02-14T17:41:31.023725+0000","last_undegraded":"2024-02-14T17:41:31.023725+0000","last_fullsized":"2024-02-14T17:41:31.023725+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:12:19.847910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023431+0000","last_change":"2024-02-14T17:41:06.401682+0000","last_active":"2024-02-14T17:41:31.023431+0000","last_peered":"2024-02-14T17:41:31.023431+0000","last_clean":"2024-02-14T17:41:31.023431+0000","last_became_active":"2024-02-14T17:41:06.400268+0000","last_became_peered":"2024-02-14T17:41:06.400268+0000","last_unstale":"2024-02-14T17:41:31.023431+0000","last_undegraded":"2024-02-14T17:41:31.023431+0000","last_fullsized":"2024-02-14T17:41:31.023431+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:24:15.778505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025322+0000","last_change":"2024-02-14T17:41:02.378302+0000","last_active":"2024-02-14T17:41:31.025322+0000","last_peered":"2024-02-14T17:41:31.025322+0000","last_clean":"2024-02-14T17:41:31.025322+0000","last_became_active":"2024-02-14T17:41:02.377857+0000","last_became_peered":"2024-02-14T17:41:02.377857+0000","last_unstale":"2024-02-14T17:41:31.025322+0000","last_undegraded":"2024-02-14T17:41:31.025322+0000","last_fullsized":"2024-02-14T17:41:31.025322+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:39:11.805991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026496+0000","last_change":"2024-02-14T17:41:04.386956+0000","last_active":"2024-02-14T17:41:31.026496+0000","last_peered":"2024-02-14T17:41:31.026496+0000","last_clean":"2024-02-14T17:41:31.026496+0000","last_became_active":"2024-02-14T17:41:04.386184+0000","last_became_peered":"2024-02-14T17:41:04.386184+0000","last_unstale":"2024-02-14T17:41:31.026496+0000","last_undegraded":"2024-02-14T17:41:31.026496+0000","last_fullsized":"2024-02-14T17:41:31.026496+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:37:54.980130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024842+0000","last_change":"2024-02-14T17:41:08.412583+0000","last_active":"2024-02-14T17:41:31.024842+0000","last_peered":"2024-02-14T17:41:31.024842+0000","last_clean":"2024-02-14T17:41:31.024842+0000","last_became_active":"2024-02-14T17:41:08.412286+0000","last_became_peered":"2024-02-14T17:41:08.412286+0000","last_unstale":"2024-02-14T17:41:31.024842+0000","last_undegraded":"2024-02-14T17:41:31.024842+0000","last_fullsized":"2024-02-14T17:41:31.024842+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:56:11.894166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024273+0000","last_change":"2024-02-14T17:41:06.400846+0000","last_active":"2024-02-14T17:41:31.024273+0000","last_peered":"2024-02-14T17:41:31.024273+0000","last_clean":"2024-02-14T17:41:31.024273+0000","last_became_active":"2024-02-14T17:41:06.400500+0000","last_became_peered":"2024-02-14T17:41:06.400500+0000","last_unstale":"2024-02-14T17:41:31.024273+0000","last_undegraded":"2024-02-14T17:41:31.024273+0000","last_fullsized":"2024-02-14T17:41:31.024273+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:44:46.320226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027855+0000","last_change":"2024-02-14T17:41:02.379380+0000","last_active":"2024-02-14T17:41:31.027855+0000","last_peered":"2024-02-14T17:41:31.027855+0000","last_clean":"2024-02-14T17:41:31.027855+0000","last_became_active":"2024-02-14T17:41:02.378234+0000","last_became_peered":"2024-02-14T17:41:02.378234+0000","last_unstale":"2024-02-14T17:41:31.027855+0000","last_undegraded":"2024-02-14T17:41:31.027855+0000","last_fullsized":"2024-02-14T17:41:31.027855+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:36:35.084518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026134+0000","last_change":"2024-02-14T17:41:04.386399+0000","last_active":"2024-02-14T17:41:31.026134+0000","last_peered":"2024-02-14T17:41:31.026134+0000","last_clean":"2024-02-14T17:41:31.026134+0000","last_became_active":"2024-02-14T17:41:04.385166+0000","last_became_peered":"2024-02-14T17:41:04.385166+0000","last_unstale":"2024-02-14T17:41:31.026134+0000","last_undegraded":"2024-02-14T17:41:31.026134+0000","last_fullsized":"2024-02-14T17:41:31.026134+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:52:15.458766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024012+0000","last_change":"2024-02-14T17:41:08.412131+0000","last_active":"2024-02-14T17:41:31.024012+0000","last_peered":"2024-02-14T17:41:31.024012+0000","last_clean":"2024-02-14T17:41:31.024012+0000","last_became_active":"2024-02-14T17:41:08.411651+0000","last_became_peered":"2024-02-14T17:41:08.411651+0000","last_unstale":"2024-02-14T17:41:31.024012+0000","last_undegraded":"2024-02-14T17:41:31.024012+0000","last_fullsized":"2024-02-14T17:41:31.024012+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:34:35.224612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023387+0000","last_change":"2024-02-14T17:41:06.401047+0000","last_active":"2024-02-14T17:41:31.023387+0000","last_peered":"2024-02-14T17:41:31.023387+0000","last_clean":"2024-02-14T17:41:31.023387+0000","last_became_active":"2024-02-14T17:41:06.400216+0000","last_became_peered":"2024-02-14T17:41:06.400216+0000","last_unstale":"2024-02-14T17:41:31.023387+0000","last_undegraded":"2024-02-14T17:41:31.023387+0000","last_fullsized":"2024-02-14T17:41:31.023387+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:44:41.178492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026742+0000","last_change":"2024-02-14T17:41:02.373621+0000","last_active":"2024-02-14T17:41:31.026742+0000","last_peered":"2024-02-14T17:41:31.026742+0000","last_clean":"2024-02-14T17:41:31.026742+0000","last_became_active":"2024-02-14T17:41:02.373205+0000","last_became_peered":"2024-02-14T17:41:02.373205+0000","last_unstale":"2024-02-14T17:41:31.026742+0000","last_undegraded":"2024-02-14T17:41:31.026742+0000","last_fullsized":"2024-02-14T17:41:31.026742+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:01:22.802544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027564+0000","last_change":"2024-02-14T17:41:04.390936+0000","last_active":"2024-02-14T17:41:31.027564+0000","last_peered":"2024-02-14T17:41:31.027564+0000","last_clean":"2024-02-14T17:41:31.027564+0000","last_became_active":"2024-02-14T17:41:04.390581+0000","last_became_peered":"2024-02-14T17:41:04.390581+0000","last_unstale":"2024-02-14T17:41:31.027564+0000","last_undegraded":"2024-02-14T17:41:31.027564+0000","last_fullsized":"2024-02-14T17:41:31.027564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:15:05.100763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025762+0000","last_change":"2024-02-14T17:41:08.412238+0000","last_active":"2024-02-14T17:41:31.025762+0000","last_peered":"2024-02-14T17:41:31.025762+0000","last_clean":"2024-02-14T17:41:31.025762+0000","last_became_active":"2024-02-14T17:41:08.411851+0000","last_became_peered":"2024-02-14T17:41:08.411851+0000","last_unstale":"2024-02-14T17:41:31.025762+0000","last_undegraded":"2024-02-14T17:41:31.025762+0000","last_fullsized":"2024-02-14T17:41:31.025762+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:14:43.981796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026687+0000","last_change":"2024-02-14T17:41:06.397548+0000","last_active":"2024-02-14T17:41:31.026687+0000","last_peered":"2024-02-14T17:41:31.026687+0000","last_clean":"2024-02-14T17:41:31.026687+0000","last_became_active":"2024-02-14T17:41:06.397263+0000","last_became_peered":"2024-02-14T17:41:06.397263+0000","last_unstale":"2024-02-14T17:41:31.026687+0000","last_undegraded":"2024-02-14T17:41:31.026687+0000","last_fullsized":"2024-02-14T17:41:31.026687+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:20:47.373218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025730+0000","last_change":"2024-02-14T17:41:02.374687+0000","last_active":"2024-02-14T17:41:31.025730+0000","last_peered":"2024-02-14T17:41:31.025730+0000","last_clean":"2024-02-14T17:41:31.025730+0000","last_became_active":"2024-02-14T17:41:02.374361+0000","last_became_peered":"2024-02-14T17:41:02.374361+0000","last_unstale":"2024-02-14T17:41:31.025730+0000","last_undegraded":"2024-02-14T17:41:31.025730+0000","last_fullsized":"2024-02-14T17:41:31.025730+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:39:56.915137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025779+0000","last_change":"2024-02-14T17:41:04.391022+0000","last_active":"2024-02-14T17:41:31.025779+0000","last_peered":"2024-02-14T17:41:31.025779+0000","last_clean":"2024-02-14T17:41:31.025779+0000","last_became_active":"2024-02-14T17:41:04.390597+0000","last_became_peered":"2024-02-14T17:41:04.390597+0000","last_unstale":"2024-02-14T17:41:31.025779+0000","last_undegraded":"2024-02-14T17:41:31.025779+0000","last_fullsized":"2024-02-14T17:41:31.025779+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T00:11:27.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025501+0000","last_change":"2024-02-14T17:41:08.411065+0000","last_active":"2024-02-14T17:41:31.025501+0000","last_peered":"2024-02-14T17:41:31.025501+0000","last_clean":"2024-02-14T17:41:31.025501+0000","last_became_active":"2024-02-14T17:41:08.410761+0000","last_became_peered":"2024-02-14T17:41:08.410761+0000","last_unstale":"2024-02-14T17:41:31.025501+0000","last_undegraded":"2024-02-14T17:41:31.025501+0000","last_fullsized":"2024-02-14T17:41:31.025501+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:28:34.606698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026657+0000","last_change":"2024-02-14T17:41:06.397434+0000","last_active":"2024-02-14T17:41:31.026657+0000","last_peered":"2024-02-14T17:41:31.026657+0000","last_clean":"2024-02-14T17:41:31.026657+0000","last_became_active":"2024-02-14T17:41:06.397004+0000","last_became_peered":"2024-02-14T17:41:06.397004+0000","last_unstale":"2024-02-14T17:41:31.026657+0000","last_undegraded":"2024-02-14T17:41:31.026657+0000","last_fullsized":"2024-02-14T17:41:31.026657+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:22:26.898910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024052+0000","last_change":"2024-02-14T17:41:02.374878+0000","last_active":"2024-02-14T17:41:31.024052+0000","last_peered":"2024-02-14T17:41:31.024052+0000","last_clean":"2024-02-14T17:41:31.024052+0000","last_became_active":"2024-02-14T17:41:02.374387+0000","last_became_peered":"2024-02-14T17:41:02.374387+0000","last_unstale":"2024-02-14T17:41:31.024052+0000","last_undegraded":"2024-02-14T17:41:31.024052+0000","last_fullsized":"2024-02-14T17:41:31.024052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:12:05.740629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025525+0000","last_change":"2024-02-14T17:41:04.391265+0000","last_active":"2024-02-14T17:41:31.025525+0000","last_peered":"2024-02-14T17:41:31.025525+0000","last_clean":"2024-02-14T17:41:31.025525+0000","last_became_active":"2024-02-14T17:41:04.390926+0000","last_became_peered":"2024-02-14T17:41:04.390926+0000","last_unstale":"2024-02-14T17:41:31.025525+0000","last_undegraded":"2024-02-14T17:41:31.025525+0000","last_fullsized":"2024-02-14T17:41:31.025525+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T00:22:23.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023953+0000","last_change":"2024-02-14T17:41:08.412098+0000","last_active":"2024-02-14T17:41:31.023953+0000","last_peered":"2024-02-14T17:41:31.023953+0000","last_clean":"2024-02-14T17:41:31.023953+0000","last_became_active":"2024-02-14T17:41:08.411698+0000","last_became_peered":"2024-02-14T17:41:08.411698+0000","last_unstale":"2024-02-14T17:41:31.023953+0000","last_undegraded":"2024-02-14T17:41:31.023953+0000","last_fullsized":"2024-02-14T17:41:31.023953+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:39:45.814581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:47.427336+0000","last_change":"2024-02-14T17:41:06.400575+0000","last_active":"2024-02-14T17:41:47.427336+0000","last_peered":"2024-02-14T17:41:47.427336+0000","last_clean":"2024-02-14T17:41:47.427336+0000","last_became_active":"2024-02-14T17:41:06.399708+0000","last_became_peered":"2024-02-14T17:41:06.399708+0000","last_unstale":"2024-02-14T17:41:47.427336+0000","last_undegraded":"2024-02-14T17:41:47.427336+0000","last_fullsized":"2024-02-14T17:41:47.427336+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:44:25.603955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024191+0000","last_change":"2024-02-14T17:41:02.375352+0000","last_active":"2024-02-14T17:41:31.024191+0000","last_peered":"2024-02-14T17:41:31.024191+0000","last_clean":"2024-02-14T17:41:31.024191+0000","last_became_active":"2024-02-14T17:41:02.374866+0000","last_became_peered":"2024-02-14T17:41:02.374866+0000","last_unstale":"2024-02-14T17:41:31.024191+0000","last_undegraded":"2024-02-14T17:41:31.024191+0000","last_fullsized":"2024-02-14T17:41:31.024191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:24:12.343347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023132+0000","last_change":"2024-02-14T17:41:04.385887+0000","last_active":"2024-02-14T17:41:31.023132+0000","last_peered":"2024-02-14T17:41:31.023132+0000","last_clean":"2024-02-14T17:41:31.023132+0000","last_became_active":"2024-02-14T17:41:04.384679+0000","last_became_peered":"2024-02-14T17:41:04.384679+0000","last_unstale":"2024-02-14T17:41:31.023132+0000","last_undegraded":"2024-02-14T17:41:31.023132+0000","last_fullsized":"2024-02-14T17:41:31.023132+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:22:17.122776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024119+0000","last_change":"2024-02-14T17:41:08.412325+0000","last_active":"2024-02-14T17:41:31.024119+0000","last_peered":"2024-02-14T17:41:31.024119+0000","last_clean":"2024-02-14T17:41:31.024119+0000","last_became_active":"2024-02-14T17:41:08.411948+0000","last_became_peered":"2024-02-14T17:41:08.411948+0000","last_unstale":"2024-02-14T17:41:31.024119+0000","last_undegraded":"2024-02-14T17:41:31.024119+0000","last_fullsized":"2024-02-14T17:41:31.024119+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:11:08.186746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024303+0000","last_change":"2024-02-14T17:41:06.398714+0000","last_active":"2024-02-14T17:41:31.024303+0000","last_peered":"2024-02-14T17:41:31.024303+0000","last_clean":"2024-02-14T17:41:31.024303+0000","last_became_active":"2024-02-14T17:41:06.398188+0000","last_became_peered":"2024-02-14T17:41:06.398188+0000","last_unstale":"2024-02-14T17:41:31.024303+0000","last_undegraded":"2024-02-14T17:41:31.024303+0000","last_fullsized":"2024-02-14T17:41:31.024303+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:04:10.150694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024783+0000","last_change":"2024-02-14T17:41:02.374502+0000","last_active":"2024-02-14T17:41:31.024783+0000","last_peered":"2024-02-14T17:41:31.024783+0000","last_clean":"2024-02-14T17:41:31.024783+0000","last_became_active":"2024-02-14T17:41:02.373525+0000","last_became_peered":"2024-02-14T17:41:02.373525+0000","last_unstale":"2024-02-14T17:41:31.024783+0000","last_undegraded":"2024-02-14T17:41:31.024783+0000","last_fullsized":"2024-02-14T17:41:31.024783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:10:26.283469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024339+0000","last_change":"2024-02-14T17:41:04.384407+0000","last_active":"2024-02-14T17:41:31.024339+0000","last_peered":"2024-02-14T17:41:31.024339+0000","last_clean":"2024-02-14T17:41:31.024339+0000","last_became_active":"2024-02-14T17:41:04.384027+0000","last_became_peered":"2024-02-14T17:41:04.384027+0000","last_unstale":"2024-02-14T17:41:31.024339+0000","last_undegraded":"2024-02-14T17:41:31.024339+0000","last_fullsized":"2024-02-14T17:41:31.024339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:18:32.190114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023649+0000","last_change":"2024-02-14T17:41:08.409448+0000","last_active":"2024-02-14T17:41:31.023649+0000","last_peered":"2024-02-14T17:41:31.023649+0000","last_clean":"2024-02-14T17:41:31.023649+0000","last_became_active":"2024-02-14T17:41:08.408388+0000","last_became_peered":"2024-02-14T17:41:08.408388+0000","last_unstale":"2024-02-14T17:41:31.023649+0000","last_undegraded":"2024-02-14T17:41:31.023649+0000","last_fullsized":"2024-02-14T17:41:31.023649+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:11:32.120421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023388+0000","last_change":"2024-02-14T17:41:06.400935+0000","last_active":"2024-02-14T17:41:31.023388+0000","last_peered":"2024-02-14T17:41:31.023388+0000","last_clean":"2024-02-14T17:41:31.023388+0000","last_became_active":"2024-02-14T17:41:06.400550+0000","last_became_peered":"2024-02-14T17:41:06.400550+0000","last_unstale":"2024-02-14T17:41:31.023388+0000","last_undegraded":"2024-02-14T17:41:31.023388+0000","last_fullsized":"2024-02-14T17:41:31.023388+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:19:57.099758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025178+0000","last_change":"2024-02-14T17:41:02.377901+0000","last_active":"2024-02-14T17:41:31.025178+0000","last_peered":"2024-02-14T17:41:31.025178+0000","last_clean":"2024-02-14T17:41:31.025178+0000","last_became_active":"2024-02-14T17:41:02.377486+0000","last_became_peered":"2024-02-14T17:41:02.377486+0000","last_unstale":"2024-02-14T17:41:31.025178+0000","last_undegraded":"2024-02-14T17:41:31.025178+0000","last_fullsized":"2024-02-14T17:41:31.025178+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:47:16.432930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025655+0000","last_change":"2024-02-14T17:41:04.384956+0000","last_active":"2024-02-14T17:41:31.025655+0000","last_peered":"2024-02-14T17:41:31.025655+0000","last_clean":"2024-02-14T17:41:31.025655+0000","last_became_active":"2024-02-14T17:41:04.384575+0000","last_became_peered":"2024-02-14T17:41:04.384575+0000","last_unstale":"2024-02-14T17:41:31.025655+0000","last_undegraded":"2024-02-14T17:41:31.025655+0000","last_fullsized":"2024-02-14T17:41:31.025655+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:48:45.219212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023612+0000","last_change":"2024-02-14T17:41:08.406762+0000","last_active":"2024-02-14T17:41:31.023612+0000","last_peered":"2024-02-14T17:41:31.023612+0000","last_clean":"2024-02-14T17:41:31.023612+0000","last_became_active":"2024-02-14T17:41:08.406418+0000","last_became_peered":"2024-02-14T17:41:08.406418+0000","last_unstale":"2024-02-14T17:41:31.023612+0000","last_undegraded":"2024-02-14T17:41:31.023612+0000","last_fullsized":"2024-02-14T17:41:31.023612+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:11:54.601485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023477+0000","last_change":"2024-02-14T17:41:06.398956+0000","last_active":"2024-02-14T17:41:31.023477+0000","last_peered":"2024-02-14T17:41:31.023477+0000","last_clean":"2024-02-14T17:41:31.023477+0000","last_became_active":"2024-02-14T17:41:06.398597+0000","last_became_peered":"2024-02-14T17:41:06.398597+0000","last_unstale":"2024-02-14T17:41:31.023477+0000","last_undegraded":"2024-02-14T17:41:31.023477+0000","last_fullsized":"2024-02-14T17:41:31.023477+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T17:46:04.813583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025208+0000","last_change":"2024-02-14T17:41:02.376868+0000","last_active":"2024-02-14T17:41:31.025208+0000","last_peered":"2024-02-14T17:41:31.025208+0000","last_clean":"2024-02-14T17:41:31.025208+0000","last_became_active":"2024-02-14T17:41:02.375350+0000","last_became_peered":"2024-02-14T17:41:02.375350+0000","last_unstale":"2024-02-14T17:41:31.025208+0000","last_undegraded":"2024-02-14T17:41:31.025208+0000","last_fullsized":"2024-02-14T17:41:31.025208+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:54:07.670987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027287+0000","last_change":"2024-02-14T17:41:04.389522+0000","last_active":"2024-02-14T17:41:31.027287+0000","last_peered":"2024-02-14T17:41:31.027287+0000","last_clean":"2024-02-14T17:41:31.027287+0000","last_became_active":"2024-02-14T17:41:04.388836+0000","last_became_peered":"2024-02-14T17:41:04.388836+0000","last_unstale":"2024-02-14T17:41:31.027287+0000","last_undegraded":"2024-02-14T17:41:31.027287+0000","last_fullsized":"2024-02-14T17:41:31.027287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:18:14.679143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025075+0000","last_change":"2024-02-14T17:41:08.412607+0000","last_active":"2024-02-14T17:41:31.025075+0000","last_peered":"2024-02-14T17:41:31.025075+0000","last_clean":"2024-02-14T17:41:31.025075+0000","last_became_active":"2024-02-14T17:41:08.412347+0000","last_became_peered":"2024-02-14T17:41:08.412347+0000","last_unstale":"2024-02-14T17:41:31.025075+0000","last_undegraded":"2024-02-14T17:41:31.025075+0000","last_fullsized":"2024-02-14T17:41:31.025075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T17:58:04.627588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024457+0000","last_change":"2024-02-14T17:41:06.401845+0000","last_active":"2024-02-14T17:41:31.024457+0000","last_peered":"2024-02-14T17:41:31.024457+0000","last_clean":"2024-02-14T17:41:31.024457+0000","last_became_active":"2024-02-14T17:41:06.401342+0000","last_became_peered":"2024-02-14T17:41:06.401342+0000","last_unstale":"2024-02-14T17:41:31.024457+0000","last_undegraded":"2024-02-14T17:41:31.024457+0000","last_fullsized":"2024-02-14T17:41:31.024457+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:26:51.292967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027871+0000","last_change":"2024-02-14T17:41:02.378515+0000","last_active":"2024-02-14T17:41:31.027871+0000","last_peered":"2024-02-14T17:41:31.027871+0000","last_clean":"2024-02-14T17:41:31.027871+0000","last_became_active":"2024-02-14T17:41:02.377309+0000","last_became_peered":"2024-02-14T17:41:02.377309+0000","last_unstale":"2024-02-14T17:41:31.027871+0000","last_undegraded":"2024-02-14T17:41:31.027871+0000","last_fullsized":"2024-02-14T17:41:31.027871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:02:04.610514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026229+0000","last_change":"2024-02-14T17:41:04.387230+0000","last_active":"2024-02-14T17:41:31.026229+0000","last_peered":"2024-02-14T17:41:31.026229+0000","last_clean":"2024-02-14T17:41:31.026229+0000","last_became_active":"2024-02-14T17:41:04.386230+0000","last_became_peered":"2024-02-14T17:41:04.386230+0000","last_unstale":"2024-02-14T17:41:31.026229+0000","last_undegraded":"2024-02-14T17:41:31.026229+0000","last_fullsized":"2024-02-14T17:41:31.026229+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:28:25.522058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025123+0000","last_change":"2024-02-14T17:41:08.412479+0000","last_active":"2024-02-14T17:41:31.025123+0000","last_peered":"2024-02-14T17:41:31.025123+0000","last_clean":"2024-02-14T17:41:31.025123+0000","last_became_active":"2024-02-14T17:41:08.412046+0000","last_became_peered":"2024-02-14T17:41:08.412046+0000","last_unstale":"2024-02-14T17:41:31.025123+0000","last_undegraded":"2024-02-14T17:41:31.025123+0000","last_fullsized":"2024-02-14T17:41:31.025123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:23:58.892388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023499+0000","last_change":"2024-02-14T17:41:06.398121+0000","last_active":"2024-02-14T17:41:31.023499+0000","last_peered":"2024-02-14T17:41:31.023499+0000","last_clean":"2024-02-14T17:41:31.023499+0000","last_became_active":"2024-02-14T17:41:06.397728+0000","last_became_peered":"2024-02-14T17:41:06.397728+0000","last_unstale":"2024-02-14T17:41:31.023499+0000","last_undegraded":"2024-02-14T17:41:31.023499+0000","last_fullsized":"2024-02-14T17:41:31.023499+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:02:11.540347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025990+0000","last_change":"2024-02-14T17:41:02.376309+0000","last_active":"2024-02-14T17:41:31.025990+0000","last_peered":"2024-02-14T17:41:31.025990+0000","last_clean":"2024-02-14T17:41:31.025990+0000","last_became_active":"2024-02-14T17:41:02.374567+0000","last_became_peered":"2024-02-14T17:41:02.374567+0000","last_unstale":"2024-02-14T17:41:31.025990+0000","last_undegraded":"2024-02-14T17:41:31.025990+0000","last_fullsized":"2024-02-14T17:41:31.025990+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:20:07.899727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027571+0000","last_change":"2024-02-14T17:41:04.391275+0000","last_active":"2024-02-14T17:41:31.027571+0000","last_peered":"2024-02-14T17:41:31.027571+0000","last_clean":"2024-02-14T17:41:31.027571+0000","last_became_active":"2024-02-14T17:41:04.390816+0000","last_became_peered":"2024-02-14T17:41:04.390816+0000","last_unstale":"2024-02-14T17:41:31.027571+0000","last_undegraded":"2024-02-14T17:41:31.027571+0000","last_fullsized":"2024-02-14T17:41:31.027571+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:24:46.327420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025567+0000","last_change":"2024-02-14T17:41:08.409579+0000","last_active":"2024-02-14T17:41:31.025567+0000","last_peered":"2024-02-14T17:41:31.025567+0000","last_clean":"2024-02-14T17:41:31.025567+0000","last_became_active":"2024-02-14T17:41:08.409224+0000","last_became_peered":"2024-02-14T17:41:08.409224+0000","last_unstale":"2024-02-14T17:41:31.025567+0000","last_undegraded":"2024-02-14T17:41:31.025567+0000","last_fullsized":"2024-02-14T17:41:31.025567+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:06:13.387021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:47.427328+0000","last_change":"2024-02-14T17:41:06.400941+0000","last_active":"2024-02-14T17:41:47.427328+0000","last_peered":"2024-02-14T17:41:47.427328+0000","last_clean":"2024-02-14T17:41:47.427328+0000","last_became_active":"2024-02-14T17:41:06.399688+0000","last_became_peered":"2024-02-14T17:41:06.399688+0000","last_unstale":"2024-02-14T17:41:47.427328+0000","last_undegraded":"2024-02-14T17:41:47.427328+0000","last_fullsized":"2024-02-14T17:41:47.427328+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:12:16.887899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025837+0000","last_change":"2024-02-14T17:41:02.374652+0000","last_active":"2024-02-14T17:41:31.025837+0000","last_peered":"2024-02-14T17:41:31.025837+0000","last_clean":"2024-02-14T17:41:31.025837+0000","last_became_active":"2024-02-14T17:41:02.373637+0000","last_became_peered":"2024-02-14T17:41:02.373637+0000","last_unstale":"2024-02-14T17:41:31.025837+0000","last_undegraded":"2024-02-14T17:41:31.025837+0000","last_fullsized":"2024-02-14T17:41:31.025837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:11:24.842191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024079+0000","last_change":"2024-02-14T17:41:04.385500+0000","last_active":"2024-02-14T17:41:31.024079+0000","last_peered":"2024-02-14T17:41:31.024079+0000","last_clean":"2024-02-14T17:41:31.024079+0000","last_became_active":"2024-02-14T17:41:04.384913+0000","last_became_peered":"2024-02-14T17:41:04.384913+0000","last_unstale":"2024-02-14T17:41:31.024079+0000","last_undegraded":"2024-02-14T17:41:31.024079+0000","last_fullsized":"2024-02-14T17:41:31.024079+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:32:47.253287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025188+0000","last_change":"2024-02-14T17:41:08.412099+0000","last_active":"2024-02-14T17:41:31.025188+0000","last_peered":"2024-02-14T17:41:31.025188+0000","last_clean":"2024-02-14T17:41:31.025188+0000","last_became_active":"2024-02-14T17:41:08.411401+0000","last_became_peered":"2024-02-14T17:41:08.411401+0000","last_unstale":"2024-02-14T17:41:31.025188+0000","last_undegraded":"2024-02-14T17:41:31.025188+0000","last_fullsized":"2024-02-14T17:41:31.025188+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:38:20.189799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:47.427310+0000","last_change":"2024-02-14T17:41:06.400291+0000","last_active":"2024-02-14T17:41:47.427310+0000","last_peered":"2024-02-14T17:41:47.427310+0000","last_clean":"2024-02-14T17:41:47.427310+0000","last_became_active":"2024-02-14T17:41:06.399919+0000","last_became_peered":"2024-02-14T17:41:06.399919+0000","last_unstale":"2024-02-14T17:41:47.427310+0000","last_undegraded":"2024-02-14T17:41:47.427310+0000","last_fullsized":"2024-02-14T17:41:47.427310+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:41:06.807684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025247+0000","last_change":"2024-02-14T17:41:02.375597+0000","last_active":"2024-02-14T17:41:31.025247+0000","last_peered":"2024-02-14T17:41:31.025247+0000","last_clean":"2024-02-14T17:41:31.025247+0000","last_became_active":"2024-02-14T17:41:02.374050+0000","last_became_peered":"2024-02-14T17:41:02.374050+0000","last_unstale":"2024-02-14T17:41:31.025247+0000","last_undegraded":"2024-02-14T17:41:31.025247+0000","last_fullsized":"2024-02-14T17:41:31.025247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:14:46.169240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024859+0000","last_change":"2024-02-14T17:41:04.384674+0000","last_active":"2024-02-14T17:41:31.024859+0000","last_peered":"2024-02-14T17:41:31.024859+0000","last_clean":"2024-02-14T17:41:31.024859+0000","last_became_active":"2024-02-14T17:41:04.384273+0000","last_became_peered":"2024-02-14T17:41:04.384273+0000","last_unstale":"2024-02-14T17:41:31.024859+0000","last_undegraded":"2024-02-14T17:41:31.024859+0000","last_fullsized":"2024-02-14T17:41:31.024859+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:59:06.854382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024614+0000","last_change":"2024-02-14T17:41:08.412707+0000","last_active":"2024-02-14T17:41:31.024614+0000","last_peered":"2024-02-14T17:41:31.024614+0000","last_clean":"2024-02-14T17:41:31.024614+0000","last_became_active":"2024-02-14T17:41:08.412329+0000","last_became_peered":"2024-02-14T17:41:08.412329+0000","last_unstale":"2024-02-14T17:41:31.024614+0000","last_undegraded":"2024-02-14T17:41:31.024614+0000","last_fullsized":"2024-02-14T17:41:31.024614+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:55:04.996507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026281+0000","last_change":"2024-02-14T17:41:06.400816+0000","last_active":"2024-02-14T17:41:31.026281+0000","last_peered":"2024-02-14T17:41:31.026281+0000","last_clean":"2024-02-14T17:41:31.026281+0000","last_became_active":"2024-02-14T17:41:06.399580+0000","last_became_peered":"2024-02-14T17:41:06.399580+0000","last_unstale":"2024-02-14T17:41:31.026281+0000","last_undegraded":"2024-02-14T17:41:31.026281+0000","last_fullsized":"2024-02-14T17:41:31.026281+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:13:32.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023804+0000","last_change":"2024-02-14T17:41:02.375123+0000","last_active":"2024-02-14T17:41:31.023804+0000","last_peered":"2024-02-14T17:41:31.023804+0000","last_clean":"2024-02-14T17:41:31.023804+0000","last_became_active":"2024-02-14T17:41:02.374470+0000","last_became_peered":"2024-02-14T17:41:02.374470+0000","last_unstale":"2024-02-14T17:41:31.023804+0000","last_undegraded":"2024-02-14T17:41:31.023804+0000","last_fullsized":"2024-02-14T17:41:31.023804+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:48:56.359866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023034+0000","last_change":"2024-02-14T17:41:04.385521+0000","last_active":"2024-02-14T17:41:31.023034+0000","last_peered":"2024-02-14T17:41:31.023034+0000","last_clean":"2024-02-14T17:41:31.023034+0000","last_became_active":"2024-02-14T17:41:04.384711+0000","last_became_peered":"2024-02-14T17:41:04.384711+0000","last_unstale":"2024-02-14T17:41:31.023034+0000","last_undegraded":"2024-02-14T17:41:31.023034+0000","last_fullsized":"2024-02-14T17:41:31.023034+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:28:46.379419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026478+0000","last_change":"2024-02-14T17:41:08.411168+0000","last_active":"2024-02-14T17:41:31.026478+0000","last_peered":"2024-02-14T17:41:31.026478+0000","last_clean":"2024-02-14T17:41:31.026478+0000","last_became_active":"2024-02-14T17:41:08.410861+0000","last_became_peered":"2024-02-14T17:41:08.410861+0000","last_unstale":"2024-02-14T17:41:31.026478+0000","last_undegraded":"2024-02-14T17:41:31.026478+0000","last_fullsized":"2024-02-14T17:41:31.026478+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:42:56.373255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024247+0000","last_change":"2024-02-14T17:41:06.399345+0000","last_active":"2024-02-14T17:41:31.024247+0000","last_peered":"2024-02-14T17:41:31.024247+0000","last_clean":"2024-02-14T17:41:31.024247+0000","last_became_active":"2024-02-14T17:41:06.398671+0000","last_became_peered":"2024-02-14T17:41:06.398671+0000","last_unstale":"2024-02-14T17:41:31.024247+0000","last_undegraded":"2024-02-14T17:41:31.024247+0000","last_fullsized":"2024-02-14T17:41:31.024247+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:27:35.100179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024838+0000","last_change":"2024-02-14T17:41:02.379654+0000","last_active":"2024-02-14T17:41:31.024838+0000","last_peered":"2024-02-14T17:41:31.024838+0000","last_clean":"2024-02-14T17:41:31.024838+0000","last_became_active":"2024-02-14T17:41:02.379195+0000","last_became_peered":"2024-02-14T17:41:02.379195+0000","last_unstale":"2024-02-14T17:41:31.024838+0000","last_undegraded":"2024-02-14T17:41:31.024838+0000","last_fullsized":"2024-02-14T17:41:31.024838+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:08:14.065622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026238+0000","last_change":"2024-02-14T17:41:04.389438+0000","last_active":"2024-02-14T17:41:31.026238+0000","last_peered":"2024-02-14T17:41:31.026238+0000","last_clean":"2024-02-14T17:41:31.026238+0000","last_became_active":"2024-02-14T17:41:04.389160+0000","last_became_peered":"2024-02-14T17:41:04.389160+0000","last_unstale":"2024-02-14T17:41:31.026238+0000","last_undegraded":"2024-02-14T17:41:31.026238+0000","last_fullsized":"2024-02-14T17:41:31.026238+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:46:55.647026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025940+0000","last_change":"2024-02-14T17:41:08.414494+0000","last_active":"2024-02-14T17:41:31.025940+0000","last_peered":"2024-02-14T17:41:31.025940+0000","last_clean":"2024-02-14T17:41:31.025940+0000","last_became_active":"2024-02-14T17:41:08.414146+0000","last_became_peered":"2024-02-14T17:41:08.414146+0000","last_unstale":"2024-02-14T17:41:31.025940+0000","last_undegraded":"2024-02-14T17:41:31.025940+0000","last_fullsized":"2024-02-14T17:41:31.025940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T17:51:52.672695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026756+0000","last_change":"2024-02-14T17:41:02.379126+0000","last_active":"2024-02-14T17:41:31.026756+0000","last_peered":"2024-02-14T17:41:31.026756+0000","last_clean":"2024-02-14T17:41:31.026756+0000","last_became_active":"2024-02-14T17:41:02.374828+0000","last_became_peered":"2024-02-14T17:41:02.374828+0000","last_unstale":"2024-02-14T17:41:31.026756+0000","last_undegraded":"2024-02-14T17:41:31.026756+0000","last_fullsized":"2024-02-14T17:41:31.026756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:48:58.272502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026198+0000","last_change":"2024-02-14T17:41:04.391336+0000","last_active":"2024-02-14T17:41:31.026198+0000","last_peered":"2024-02-14T17:41:31.026198+0000","last_clean":"2024-02-14T17:41:31.026198+0000","last_became_active":"2024-02-14T17:41:04.390694+0000","last_became_peered":"2024-02-14T17:41:04.390694+0000","last_unstale":"2024-02-14T17:41:31.026198+0000","last_undegraded":"2024-02-14T17:41:31.026198+0000","last_fullsized":"2024-02-14T17:41:31.026198+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:55:21.394060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:47.427363+0000","last_change":"2024-02-14T17:41:06.398177+0000","last_active":"2024-02-14T17:41:47.427363+0000","last_peered":"2024-02-14T17:41:47.427363+0000","last_clean":"2024-02-14T17:41:47.427363+0000","last_became_active":"2024-02-14T17:41:06.397559+0000","last_became_peered":"2024-02-14T17:41:06.397559+0000","last_unstale":"2024-02-14T17:41:47.427363+0000","last_undegraded":"2024-02-14T17:41:47.427363+0000","last_fullsized":"2024-02-14T17:41:47.427363+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:13:13.900330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024910+0000","last_change":"2024-02-14T17:41:08.407061+0000","last_active":"2024-02-14T17:41:31.024910+0000","last_peered":"2024-02-14T17:41:31.024910+0000","last_clean":"2024-02-14T17:41:31.024910+0000","last_became_active":"2024-02-14T17:41:08.406666+0000","last_became_peered":"2024-02-14T17:41:08.406666+0000","last_unstale":"2024-02-14T17:41:31.024910+0000","last_undegraded":"2024-02-14T17:41:31.024910+0000","last_fullsized":"2024-02-14T17:41:31.024910+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:50:42.156106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026367+0000","last_change":"2024-02-14T17:41:02.376978+0000","last_active":"2024-02-14T17:41:31.026367+0000","last_peered":"2024-02-14T17:41:31.026367+0000","last_clean":"2024-02-14T17:41:31.026367+0000","last_became_active":"2024-02-14T17:41:02.376670+0000","last_became_peered":"2024-02-14T17:41:02.376670+0000","last_unstale":"2024-02-14T17:41:31.026367+0000","last_undegraded":"2024-02-14T17:41:31.026367+0000","last_fullsized":"2024-02-14T17:41:31.026367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:27:33.525258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:47.427147+0000","last_change":"2024-02-14T17:41:04.385903+0000","last_active":"2024-02-14T17:41:47.427147+0000","last_peered":"2024-02-14T17:41:47.427147+0000","last_clean":"2024-02-14T17:41:47.427147+0000","last_became_active":"2024-02-14T17:41:04.384273+0000","last_became_peered":"2024-02-14T17:41:04.384273+0000","last_unstale":"2024-02-14T17:41:47.427147+0000","last_undegraded":"2024-02-14T17:41:47.427147+0000","last_fullsized":"2024-02-14T17:41:47.427147+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:21:21.135733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025141+0000","last_change":"2024-02-14T17:41:06.397116+0000","last_active":"2024-02-14T17:41:31.025141+0000","last_peered":"2024-02-14T17:41:31.025141+0000","last_clean":"2024-02-14T17:41:31.025141+0000","last_became_active":"2024-02-14T17:41:06.396786+0000","last_became_peered":"2024-02-14T17:41:06.396786+0000","last_unstale":"2024-02-14T17:41:31.025141+0000","last_undegraded":"2024-02-14T17:41:31.025141+0000","last_fullsized":"2024-02-14T17:41:31.025141+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:05:19.292189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025051+0000","last_change":"2024-02-14T17:41:08.410968+0000","last_active":"2024-02-14T17:41:31.025051+0000","last_peered":"2024-02-14T17:41:31.025051+0000","last_clean":"2024-02-14T17:41:31.025051+0000","last_became_active":"2024-02-14T17:41:08.410678+0000","last_became_peered":"2024-02-14T17:41:08.410678+0000","last_unstale":"2024-02-14T17:41:31.025051+0000","last_undegraded":"2024-02-14T17:41:31.025051+0000","last_fullsized":"2024-02-14T17:41:31.025051+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:44:36.538499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024219+0000","last_change":"2024-02-14T17:41:02.375584+0000","last_active":"2024-02-14T17:41:31.024219+0000","last_peered":"2024-02-14T17:41:31.024219+0000","last_clean":"2024-02-14T17:41:31.024219+0000","last_became_active":"2024-02-14T17:41:02.374702+0000","last_became_peered":"2024-02-14T17:41:02.374702+0000","last_unstale":"2024-02-14T17:41:31.024219+0000","last_undegraded":"2024-02-14T17:41:31.024219+0000","last_fullsized":"2024-02-14T17:41:31.024219+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:48:59.956150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024988+0000","last_change":"2024-02-14T17:41:04.384796+0000","last_active":"2024-02-14T17:41:31.024988+0000","last_peered":"2024-02-14T17:41:31.024988+0000","last_clean":"2024-02-14T17:41:31.024988+0000","last_became_active":"2024-02-14T17:41:04.384460+0000","last_became_peered":"2024-02-14T17:41:04.384460+0000","last_unstale":"2024-02-14T17:41:31.024988+0000","last_undegraded":"2024-02-14T17:41:31.024988+0000","last_fullsized":"2024-02-14T17:41:31.024988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:21:22.165443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025678+0000","last_change":"2024-02-14T17:41:06.399414+0000","last_active":"2024-02-14T17:41:31.025678+0000","last_peered":"2024-02-14T17:41:31.025678+0000","last_clean":"2024-02-14T17:41:31.025678+0000","last_became_active":"2024-02-14T17:41:06.398893+0000","last_became_peered":"2024-02-14T17:41:06.398893+0000","last_unstale":"2024-02-14T17:41:31.025678+0000","last_undegraded":"2024-02-14T17:41:31.025678+0000","last_fullsized":"2024-02-14T17:41:31.025678+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:32:18.973956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025605+0000","last_change":"2024-02-14T17:41:08.412220+0000","last_active":"2024-02-14T17:41:31.025605+0000","last_peered":"2024-02-14T17:41:31.025605+0000","last_clean":"2024-02-14T17:41:31.025605+0000","last_became_active":"2024-02-14T17:41:08.411861+0000","last_became_peered":"2024-02-14T17:41:08.411861+0000","last_unstale":"2024-02-14T17:41:31.025605+0000","last_undegraded":"2024-02-14T17:41:31.025605+0000","last_fullsized":"2024-02-14T17:41:31.025605+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:50:49.759434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024971+0000","last_change":"2024-02-14T17:41:04.391691+0000","last_active":"2024-02-14T17:41:31.024971+0000","last_peered":"2024-02-14T17:41:31.024971+0000","last_clean":"2024-02-14T17:41:31.024971+0000","last_became_active":"2024-02-14T17:41:04.390481+0000","last_became_peered":"2024-02-14T17:41:04.390481+0000","last_unstale":"2024-02-14T17:41:31.024971+0000","last_undegraded":"2024-02-14T17:41:31.024971+0000","last_fullsized":"2024-02-14T17:41:31.024971+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:07:37.272279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023608+0000","last_change":"2024-02-14T17:41:02.378514+0000","last_active":"2024-02-14T17:41:31.023608+0000","last_peered":"2024-02-14T17:41:31.023608+0000","last_clean":"2024-02-14T17:41:31.023608+0000","last_became_active":"2024-02-14T17:41:02.378164+0000","last_became_peered":"2024-02-14T17:41:02.378164+0000","last_unstale":"2024-02-14T17:41:31.023608+0000","last_undegraded":"2024-02-14T17:41:31.023608+0000","last_fullsized":"2024-02-14T17:41:31.023608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:32:02.997296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027217+0000","last_change":"2024-02-14T17:41:06.402091+0000","last_active":"2024-02-14T17:41:31.027217+0000","last_peered":"2024-02-14T17:41:31.027217+0000","last_clean":"2024-02-14T17:41:31.027217+0000","last_became_active":"2024-02-14T17:41:06.401691+0000","last_became_peered":"2024-02-14T17:41:06.401691+0000","last_unstale":"2024-02-14T17:41:31.027217+0000","last_undegraded":"2024-02-14T17:41:31.027217+0000","last_fullsized":"2024-02-14T17:41:31.027217+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T05:39:34.932877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025437+0000","last_change":"2024-02-14T17:41:08.412755+0000","last_active":"2024-02-14T17:41:31.025437+0000","last_peered":"2024-02-14T17:41:31.025437+0000","last_clean":"2024-02-14T17:41:31.025437+0000","last_became_active":"2024-02-14T17:41:08.412244+0000","last_became_peered":"2024-02-14T17:41:08.412244+0000","last_unstale":"2024-02-14T17:41:31.025437+0000","last_undegraded":"2024-02-14T17:41:31.025437+0000","last_fullsized":"2024-02-14T17:41:31.025437+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:01:12.104068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023881+0000","last_change":"2024-02-14T17:41:04.386897+0000","last_active":"2024-02-14T17:41:31.023881+0000","last_peered":"2024-02-14T17:41:31.023881+0000","last_clean":"2024-02-14T17:41:31.023881+0000","last_became_active":"2024-02-14T17:41:04.386157+0000","last_became_peered":"2024-02-14T17:41:04.386157+0000","last_unstale":"2024-02-14T17:41:31.023881+0000","last_undegraded":"2024-02-14T17:41:31.023881+0000","last_fullsized":"2024-02-14T17:41:31.023881+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:01:29.518979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024698+0000","last_change":"2024-02-14T17:41:02.376322+0000","last_active":"2024-02-14T17:41:31.024698+0000","last_peered":"2024-02-14T17:41:31.024698+0000","last_clean":"2024-02-14T17:41:31.024698+0000","last_became_active":"2024-02-14T17:41:02.375044+0000","last_became_peered":"2024-02-14T17:41:02.375044+0000","last_unstale":"2024-02-14T17:41:31.024698+0000","last_undegraded":"2024-02-14T17:41:31.024698+0000","last_fullsized":"2024-02-14T17:41:31.024698+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:02:55.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:33.127633+0000","last_change":"2024-02-14T17:40:49.993958+0000","last_active":"2024-02-14T17:41:33.127633+0000","last_peered":"2024-02-14T17:41:33.127633+0000","last_clean":"2024-02-14T17:41:33.127633+0000","last_became_active":"2024-02-14T17:40:49.993641+0000","last_became_peered":"2024-02-14T17:40:49.993641+0000","last_unstale":"2024-02-14T17:41:33.127633+0000","last_undegraded":"2024-02-14T17:41:33.127633+0000","last_fullsized":"2024-02-14T17:41:33.127633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:40:48.979634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:40:48.979634+0000","last_clean_scrub_stamp":"2024-02-14T17:40:48.979634+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:27:32.829382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025947+0000","last_change":"2024-02-14T17:41:06.401781+0000","last_active":"2024-02-14T17:41:31.025947+0000","last_peered":"2024-02-14T17:41:31.025947+0000","last_clean":"2024-02-14T17:41:31.025947+0000","last_became_active":"2024-02-14T17:41:06.401458+0000","last_became_peered":"2024-02-14T17:41:06.401458+0000","last_unstale":"2024-02-14T17:41:31.025947+0000","last_undegraded":"2024-02-14T17:41:31.025947+0000","last_fullsized":"2024-02-14T17:41:31.025947+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:56:20.668201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024361+0000","last_change":"2024-02-14T17:41:08.411284+0000","last_active":"2024-02-14T17:41:31.024361+0000","last_peered":"2024-02-14T17:41:31.024361+0000","last_clean":"2024-02-14T17:41:31.024361+0000","last_became_active":"2024-02-14T17:41:08.409981+0000","last_became_peered":"2024-02-14T17:41:08.409981+0000","last_unstale":"2024-02-14T17:41:31.024361+0000","last_undegraded":"2024-02-14T17:41:31.024361+0000","last_fullsized":"2024-02-14T17:41:31.024361+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:57:18.029925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025054+0000","last_change":"2024-02-14T17:41:04.385719+0000","last_active":"2024-02-14T17:41:31.025054+0000","last_peered":"2024-02-14T17:41:31.025054+0000","last_clean":"2024-02-14T17:41:31.025054+0000","last_became_active":"2024-02-14T17:41:04.385254+0000","last_became_peered":"2024-02-14T17:41:04.385254+0000","last_unstale":"2024-02-14T17:41:31.025054+0000","last_undegraded":"2024-02-14T17:41:31.025054+0000","last_fullsized":"2024-02-14T17:41:31.025054+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:16:28.148386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023062+0000","last_change":"2024-02-14T17:41:02.376882+0000","last_active":"2024-02-14T17:41:31.023062+0000","last_peered":"2024-02-14T17:41:31.023062+0000","last_clean":"2024-02-14T17:41:31.023062+0000","last_became_active":"2024-02-14T17:41:02.375178+0000","last_became_peered":"2024-02-14T17:41:02.375178+0000","last_unstale":"2024-02-14T17:41:31.023062+0000","last_undegraded":"2024-02-14T17:41:31.023062+0000","last_fullsized":"2024-02-14T17:41:31.023062+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:27:45.838111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025977+0000","last_change":"2024-02-14T17:41:06.398222+0000","last_active":"2024-02-14T17:41:31.025977+0000","last_peered":"2024-02-14T17:41:31.025977+0000","last_clean":"2024-02-14T17:41:31.025977+0000","last_became_active":"2024-02-14T17:41:06.397673+0000","last_became_peered":"2024-02-14T17:41:06.397673+0000","last_unstale":"2024-02-14T17:41:31.025977+0000","last_undegraded":"2024-02-14T17:41:31.025977+0000","last_fullsized":"2024-02-14T17:41:31.025977+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:01:28.963313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023939+0000","last_change":"2024-02-14T17:41:08.408359+0000","last_active":"2024-02-14T17:41:31.023939+0000","last_peered":"2024-02-14T17:41:31.023939+0000","last_clean":"2024-02-14T17:41:31.023939+0000","last_became_active":"2024-02-14T17:41:08.407915+0000","last_became_peered":"2024-02-14T17:41:08.407915+0000","last_unstale":"2024-02-14T17:41:31.023939+0000","last_undegraded":"2024-02-14T17:41:31.023939+0000","last_fullsized":"2024-02-14T17:41:31.023939+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:42:45.553065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027783+0000","last_change":"2024-02-14T17:41:02.379121+0000","last_active":"2024-02-14T17:41:31.027783+0000","last_peered":"2024-02-14T17:41:31.027783+0000","last_clean":"2024-02-14T17:41:31.027783+0000","last_became_active":"2024-02-14T17:41:02.377718+0000","last_became_peered":"2024-02-14T17:41:02.377718+0000","last_unstale":"2024-02-14T17:41:31.027783+0000","last_undegraded":"2024-02-14T17:41:31.027783+0000","last_fullsized":"2024-02-14T17:41:31.027783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:45:49.190070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:47.427091+0000","last_change":"2024-02-14T17:41:04.387001+0000","last_active":"2024-02-14T17:41:47.427091+0000","last_peered":"2024-02-14T17:41:47.427091+0000","last_clean":"2024-02-14T17:41:47.427091+0000","last_became_active":"2024-02-14T17:41:04.386247+0000","last_became_peered":"2024-02-14T17:41:04.386247+0000","last_unstale":"2024-02-14T17:41:47.427091+0000","last_undegraded":"2024-02-14T17:41:47.427091+0000","last_fullsized":"2024-02-14T17:41:47.427091+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:12:23.487876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:47.427350+0000","last_change":"2024-02-14T17:41:06.397799+0000","last_active":"2024-02-14T17:41:47.427350+0000","last_peered":"2024-02-14T17:41:47.427350+0000","last_clean":"2024-02-14T17:41:47.427350+0000","last_became_active":"2024-02-14T17:41:06.397505+0000","last_became_peered":"2024-02-14T17:41:06.397505+0000","last_unstale":"2024-02-14T17:41:47.427350+0000","last_undegraded":"2024-02-14T17:41:47.427350+0000","last_fullsized":"2024-02-14T17:41:47.427350+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:46:53.143729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023142+0000","last_change":"2024-02-14T17:41:08.410153+0000","last_active":"2024-02-14T17:41:31.023142+0000","last_peered":"2024-02-14T17:41:31.023142+0000","last_clean":"2024-02-14T17:41:31.023142+0000","last_became_active":"2024-02-14T17:41:08.409629+0000","last_became_peered":"2024-02-14T17:41:08.409629+0000","last_unstale":"2024-02-14T17:41:31.023142+0000","last_undegraded":"2024-02-14T17:41:31.023142+0000","last_fullsized":"2024-02-14T17:41:31.023142+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T17:46:34.678202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025497+0000","last_change":"2024-02-14T17:41:02.378465+0000","last_active":"2024-02-14T17:41:31.025497+0000","last_peered":"2024-02-14T17:41:31.025497+0000","last_clean":"2024-02-14T17:41:31.025497+0000","last_became_active":"2024-02-14T17:41:02.378118+0000","last_became_peered":"2024-02-14T17:41:02.378118+0000","last_unstale":"2024-02-14T17:41:31.025497+0000","last_undegraded":"2024-02-14T17:41:31.025497+0000","last_fullsized":"2024-02-14T17:41:31.025497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:07:09.904389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027470+0000","last_change":"2024-02-14T17:41:04.389601+0000","last_active":"2024-02-14T17:41:31.027470+0000","last_peered":"2024-02-14T17:41:31.027470+0000","last_clean":"2024-02-14T17:41:31.027470+0000","last_became_active":"2024-02-14T17:41:04.389000+0000","last_became_peered":"2024-02-14T17:41:04.389000+0000","last_unstale":"2024-02-14T17:41:31.027470+0000","last_undegraded":"2024-02-14T17:41:31.027470+0000","last_fullsized":"2024-02-14T17:41:31.027470+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:21:05.973368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:47.427027+0000","last_change":"2024-02-14T17:41:06.398490+0000","last_active":"2024-02-14T17:41:47.427027+0000","last_peered":"2024-02-14T17:41:47.427027+0000","last_clean":"2024-02-14T17:41:47.427027+0000","last_became_active":"2024-02-14T17:41:06.397851+0000","last_became_peered":"2024-02-14T17:41:06.397851+0000","last_unstale":"2024-02-14T17:41:47.427027+0000","last_undegraded":"2024-02-14T17:41:47.427027+0000","last_fullsized":"2024-02-14T17:41:47.427027+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:58:32.448296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025848+0000","last_change":"2024-02-14T17:41:08.413803+0000","last_active":"2024-02-14T17:41:31.025848+0000","last_peered":"2024-02-14T17:41:31.025848+0000","last_clean":"2024-02-14T17:41:31.025848+0000","last_became_active":"2024-02-14T17:41:08.413410+0000","last_became_peered":"2024-02-14T17:41:08.413410+0000","last_unstale":"2024-02-14T17:41:31.025848+0000","last_undegraded":"2024-02-14T17:41:31.025848+0000","last_fullsized":"2024-02-14T17:41:31.025848+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:01:10.293991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024655+0000","last_change":"2024-02-14T17:41:02.376920+0000","last_active":"2024-02-14T17:41:31.024655+0000","last_peered":"2024-02-14T17:41:31.024655+0000","last_clean":"2024-02-14T17:41:31.024655+0000","last_became_active":"2024-02-14T17:41:02.376259+0000","last_became_peered":"2024-02-14T17:41:02.376259+0000","last_unstale":"2024-02-14T17:41:31.024655+0000","last_undegraded":"2024-02-14T17:41:31.024655+0000","last_fullsized":"2024-02-14T17:41:31.024655+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:03:42.033670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025123+0000","last_change":"2024-02-14T17:41:04.385593+0000","last_active":"2024-02-14T17:41:31.025123+0000","last_peered":"2024-02-14T17:41:31.025123+0000","last_clean":"2024-02-14T17:41:31.025123+0000","last_became_active":"2024-02-14T17:41:04.384985+0000","last_became_peered":"2024-02-14T17:41:04.384985+0000","last_unstale":"2024-02-14T17:41:31.025123+0000","last_undegraded":"2024-02-14T17:41:31.025123+0000","last_fullsized":"2024-02-14T17:41:31.025123+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:43:15.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024964+0000","last_change":"2024-02-14T17:41:06.398676+0000","last_active":"2024-02-14T17:41:31.024964+0000","last_peered":"2024-02-14T17:41:31.024964+0000","last_clean":"2024-02-14T17:41:31.024964+0000","last_became_active":"2024-02-14T17:41:06.398446+0000","last_became_peered":"2024-02-14T17:41:06.398446+0000","last_unstale":"2024-02-14T17:41:31.024964+0000","last_undegraded":"2024-02-14T17:41:31.024964+0000","last_fullsized":"2024-02-14T17:41:31.024964+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:23:49.110802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024186+0000","last_change":"2024-02-14T17:41:08.412695+0000","last_active":"2024-02-14T17:41:31.024186+0000","last_peered":"2024-02-14T17:41:31.024186+0000","last_clean":"2024-02-14T17:41:31.024186+0000","last_became_active":"2024-02-14T17:41:08.412307+0000","last_became_peered":"2024-02-14T17:41:08.412307+0000","last_unstale":"2024-02-14T17:41:31.024186+0000","last_undegraded":"2024-02-14T17:41:31.024186+0000","last_fullsized":"2024-02-14T17:41:31.024186+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:07:51.183677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:47.427180+0000","last_change":"2024-02-14T17:41:06.404301+0000","last_active":"2024-02-14T17:41:47.427180+0000","last_peered":"2024-02-14T17:41:47.427180+0000","last_clean":"2024-02-14T17:41:47.427180+0000","last_became_active":"2024-02-14T17:41:06.403971+0000","last_became_peered":"2024-02-14T17:41:06.403971+0000","last_unstale":"2024-02-14T17:41:47.427180+0000","last_undegraded":"2024-02-14T17:41:47.427180+0000","last_fullsized":"2024-02-14T17:41:47.427180+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:53:58.588560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023234+0000","last_change":"2024-02-14T17:41:02.377863+0000","last_active":"2024-02-14T17:41:31.023234+0000","last_peered":"2024-02-14T17:41:31.023234+0000","last_clean":"2024-02-14T17:41:31.023234+0000","last_became_active":"2024-02-14T17:41:02.377557+0000","last_became_peered":"2024-02-14T17:41:02.377557+0000","last_unstale":"2024-02-14T17:41:31.023234+0000","last_undegraded":"2024-02-14T17:41:31.023234+0000","last_fullsized":"2024-02-14T17:41:31.023234+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:13:27.575502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026557+0000","last_change":"2024-02-14T17:41:04.391728+0000","last_active":"2024-02-14T17:41:31.026557+0000","last_peered":"2024-02-14T17:41:31.026557+0000","last_clean":"2024-02-14T17:41:31.026557+0000","last_became_active":"2024-02-14T17:41:04.391238+0000","last_became_peered":"2024-02-14T17:41:04.391238+0000","last_unstale":"2024-02-14T17:41:31.026557+0000","last_undegraded":"2024-02-14T17:41:31.026557+0000","last_fullsized":"2024-02-14T17:41:31.026557+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:34:49.987772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026345+0000","last_change":"2024-02-14T17:41:08.413366+0000","last_active":"2024-02-14T17:41:31.026345+0000","last_peered":"2024-02-14T17:41:31.026345+0000","last_clean":"2024-02-14T17:41:31.026345+0000","last_became_active":"2024-02-14T17:41:08.413097+0000","last_became_peered":"2024-02-14T17:41:08.413097+0000","last_unstale":"2024-02-14T17:41:31.026345+0000","last_undegraded":"2024-02-14T17:41:31.026345+0000","last_fullsized":"2024-02-14T17:41:31.026345+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:09:05.406307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026378+0000","last_change":"2024-02-14T17:41:06.401347+0000","last_active":"2024-02-14T17:41:31.026378+0000","last_peered":"2024-02-14T17:41:31.026378+0000","last_clean":"2024-02-14T17:41:31.026378+0000","last_became_active":"2024-02-14T17:41:06.400975+0000","last_became_peered":"2024-02-14T17:41:06.400975+0000","last_unstale":"2024-02-14T17:41:31.026378+0000","last_undegraded":"2024-02-14T17:41:31.026378+0000","last_fullsized":"2024-02-14T17:41:31.026378+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:05:03.924070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023832+0000","last_change":"2024-02-14T17:41:02.376245+0000","last_active":"2024-02-14T17:41:31.023832+0000","last_peered":"2024-02-14T17:41:31.023832+0000","last_clean":"2024-02-14T17:41:31.023832+0000","last_became_active":"2024-02-14T17:41:02.375782+0000","last_became_peered":"2024-02-14T17:41:02.375782+0000","last_unstale":"2024-02-14T17:41:31.023832+0000","last_undegraded":"2024-02-14T17:41:31.023832+0000","last_fullsized":"2024-02-14T17:41:31.023832+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:06:01.698681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025051+0000","last_change":"2024-02-14T17:41:04.391987+0000","last_active":"2024-02-14T17:41:31.025051+0000","last_peered":"2024-02-14T17:41:31.025051+0000","last_clean":"2024-02-14T17:41:31.025051+0000","last_became_active":"2024-02-14T17:41:04.391459+0000","last_became_peered":"2024-02-14T17:41:04.391459+0000","last_unstale":"2024-02-14T17:41:31.025051+0000","last_undegraded":"2024-02-14T17:41:31.025051+0000","last_fullsized":"2024-02-14T17:41:31.025051+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:08:54.650126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026567+0000","last_change":"2024-02-14T17:41:08.413463+0000","last_active":"2024-02-14T17:41:31.026567+0000","last_peered":"2024-02-14T17:41:31.026567+0000","last_clean":"2024-02-14T17:41:31.026567+0000","last_became_active":"2024-02-14T17:41:08.413167+0000","last_became_peered":"2024-02-14T17:41:08.413167+0000","last_unstale":"2024-02-14T17:41:31.026567+0000","last_undegraded":"2024-02-14T17:41:31.026567+0000","last_fullsized":"2024-02-14T17:41:31.026567+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:15:20.187431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025679+0000","last_change":"2024-02-14T17:41:02.377592+0000","last_active":"2024-02-14T17:41:31.025679+0000","last_peered":"2024-02-14T17:41:31.025679+0000","last_clean":"2024-02-14T17:41:31.025679+0000","last_became_active":"2024-02-14T17:41:02.377236+0000","last_became_peered":"2024-02-14T17:41:02.377236+0000","last_unstale":"2024-02-14T17:41:31.025679+0000","last_undegraded":"2024-02-14T17:41:31.025679+0000","last_fullsized":"2024-02-14T17:41:31.025679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:16:15.336245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024156+0000","last_change":"2024-02-14T17:41:04.384460+0000","last_active":"2024-02-14T17:41:31.024156+0000","last_peered":"2024-02-14T17:41:31.024156+0000","last_clean":"2024-02-14T17:41:31.024156+0000","last_became_active":"2024-02-14T17:41:04.384105+0000","last_became_peered":"2024-02-14T17:41:04.384105+0000","last_unstale":"2024-02-14T17:41:31.024156+0000","last_undegraded":"2024-02-14T17:41:31.024156+0000","last_fullsized":"2024-02-14T17:41:31.024156+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:44:48.276948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025807+0000","last_change":"2024-02-14T17:41:06.401122+0000","last_active":"2024-02-14T17:41:31.025807+0000","last_peered":"2024-02-14T17:41:31.025807+0000","last_clean":"2024-02-14T17:41:31.025807+0000","last_became_active":"2024-02-14T17:41:06.400537+0000","last_became_peered":"2024-02-14T17:41:06.400537+0000","last_unstale":"2024-02-14T17:41:31.025807+0000","last_undegraded":"2024-02-14T17:41:31.025807+0000","last_fullsized":"2024-02-14T17:41:31.025807+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T04:20:35.611589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023660+0000","last_change":"2024-02-14T17:41:08.409598+0000","last_active":"2024-02-14T17:41:31.023660+0000","last_peered":"2024-02-14T17:41:31.023660+0000","last_clean":"2024-02-14T17:41:31.023660+0000","last_became_active":"2024-02-14T17:41:08.409348+0000","last_became_peered":"2024-02-14T17:41:08.409348+0000","last_unstale":"2024-02-14T17:41:31.023660+0000","last_undegraded":"2024-02-14T17:41:31.023660+0000","last_fullsized":"2024-02-14T17:41:31.023660+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T05:19:02.194386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:47.427174+0000","last_change":"2024-02-14T17:41:06.401108+0000","last_active":"2024-02-14T17:41:47.427174+0000","last_peered":"2024-02-14T17:41:47.427174+0000","last_clean":"2024-02-14T17:41:47.427174+0000","last_became_active":"2024-02-14T17:41:06.400711+0000","last_became_peered":"2024-02-14T17:41:06.400711+0000","last_unstale":"2024-02-14T17:41:47.427174+0000","last_undegraded":"2024-02-14T17:41:47.427174+0000","last_fullsized":"2024-02-14T17:41:47.427174+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:22:11.094830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025639+0000","last_change":"2024-02-14T17:41:02.375129+0000","last_active":"2024-02-14T17:41:31.025639+0000","last_peered":"2024-02-14T17:41:31.025639+0000","last_clean":"2024-02-14T17:41:31.025639+0000","last_became_active":"2024-02-14T17:41:02.374404+0000","last_became_peered":"2024-02-14T17:41:02.374404+0000","last_unstale":"2024-02-14T17:41:31.025639+0000","last_undegraded":"2024-02-14T17:41:31.025639+0000","last_fullsized":"2024-02-14T17:41:31.025639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T03:28:47.630254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024400+0000","last_change":"2024-02-14T17:41:04.387403+0000","last_active":"2024-02-14T17:41:31.024400+0000","last_peered":"2024-02-14T17:41:31.024400+0000","last_clean":"2024-02-14T17:41:31.024400+0000","last_became_active":"2024-02-14T17:41:04.386110+0000","last_became_peered":"2024-02-14T17:41:04.386110+0000","last_unstale":"2024-02-14T17:41:31.024400+0000","last_undegraded":"2024-02-14T17:41:31.024400+0000","last_fullsized":"2024-02-14T17:41:31.024400+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:46:24.866488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025354+0000","last_change":"2024-02-14T17:41:08.412661+0000","last_active":"2024-02-14T17:41:31.025354+0000","last_peered":"2024-02-14T17:41:31.025354+0000","last_clean":"2024-02-14T17:41:31.025354+0000","last_became_active":"2024-02-14T17:41:08.412353+0000","last_became_peered":"2024-02-14T17:41:08.412353+0000","last_unstale":"2024-02-14T17:41:31.025354+0000","last_undegraded":"2024-02-14T17:41:31.025354+0000","last_fullsized":"2024-02-14T17:41:31.025354+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:04:45.200992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027376+0000","last_change":"2024-02-14T17:41:02.374480+0000","last_active":"2024-02-14T17:41:31.027376+0000","last_peered":"2024-02-14T17:41:31.027376+0000","last_clean":"2024-02-14T17:41:31.027376+0000","last_became_active":"2024-02-14T17:41:02.374194+0000","last_became_peered":"2024-02-14T17:41:02.374194+0000","last_unstale":"2024-02-14T17:41:31.027376+0000","last_undegraded":"2024-02-14T17:41:31.027376+0000","last_fullsized":"2024-02-14T17:41:31.027376+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:05:50.393961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.026158+0000","last_change":"2024-02-14T17:41:04.388822+0000","last_active":"2024-02-14T17:41:31.026158+0000","last_peered":"2024-02-14T17:41:31.026158+0000","last_clean":"2024-02-14T17:41:31.026158+0000","last_became_active":"2024-02-14T17:41:04.386893+0000","last_became_peered":"2024-02-14T17:41:04.386893+0000","last_unstale":"2024-02-14T17:41:31.026158+0000","last_undegraded":"2024-02-14T17:41:31.026158+0000","last_fullsized":"2024-02-14T17:41:31.026158+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:28:55.892837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023770+0000","last_change":"2024-02-14T17:41:06.402027+0000","last_active":"2024-02-14T17:41:31.023770+0000","last_peered":"2024-02-14T17:41:31.023770+0000","last_clean":"2024-02-14T17:41:31.023770+0000","last_became_active":"2024-02-14T17:41:06.401623+0000","last_became_peered":"2024-02-14T17:41:06.401623+0000","last_unstale":"2024-02-14T17:41:31.023770+0000","last_undegraded":"2024-02-14T17:41:31.023770+0000","last_fullsized":"2024-02-14T17:41:31.023770+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:15:21.861943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023231+0000","last_change":"2024-02-14T17:41:08.407146+0000","last_active":"2024-02-14T17:41:31.023231+0000","last_peered":"2024-02-14T17:41:31.023231+0000","last_clean":"2024-02-14T17:41:31.023231+0000","last_became_active":"2024-02-14T17:41:08.406753+0000","last_became_peered":"2024-02-14T17:41:08.406753+0000","last_unstale":"2024-02-14T17:41:31.023231+0000","last_undegraded":"2024-02-14T17:41:31.023231+0000","last_fullsized":"2024-02-14T17:41:31.023231+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T23:17:12.827760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.027691+0000","last_change":"2024-02-14T17:41:02.379459+0000","last_active":"2024-02-14T17:41:31.027691+0000","last_peered":"2024-02-14T17:41:31.027691+0000","last_clean":"2024-02-14T17:41:31.027691+0000","last_became_active":"2024-02-14T17:41:02.379144+0000","last_became_peered":"2024-02-14T17:41:02.379144+0000","last_unstale":"2024-02-14T17:41:31.027691+0000","last_undegraded":"2024-02-14T17:41:31.027691+0000","last_fullsized":"2024-02-14T17:41:31.027691+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T22:51:37.479914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025999+0000","last_change":"2024-02-14T17:41:04.389949+0000","last_active":"2024-02-14T17:41:31.025999+0000","last_peered":"2024-02-14T17:41:31.025999+0000","last_clean":"2024-02-14T17:41:31.025999+0000","last_became_active":"2024-02-14T17:41:04.389386+0000","last_became_peered":"2024-02-14T17:41:04.389386+0000","last_unstale":"2024-02-14T17:41:31.025999+0000","last_undegraded":"2024-02-14T17:41:31.025999+0000","last_fullsized":"2024-02-14T17:41:31.025999+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:54:29.615217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.023802+0000","last_change":"2024-02-14T17:41:06.399367+0000","last_active":"2024-02-14T17:41:31.023802+0000","last_peered":"2024-02-14T17:41:31.023802+0000","last_clean":"2024-02-14T17:41:31.023802+0000","last_became_active":"2024-02-14T17:41:06.398644+0000","last_became_peered":"2024-02-14T17:41:06.398644+0000","last_unstale":"2024-02-14T17:41:31.023802+0000","last_undegraded":"2024-02-14T17:41:31.023802+0000","last_fullsized":"2024-02-14T17:41:31.023802+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T18:27:01.293159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.022983+0000","last_change":"2024-02-14T17:41:08.410044+0000","last_active":"2024-02-14T17:41:31.022983+0000","last_peered":"2024-02-14T17:41:31.022983+0000","last_clean":"2024-02-14T17:41:31.022983+0000","last_became_active":"2024-02-14T17:41:08.409521+0000","last_became_peered":"2024-02-14T17:41:08.409521+0000","last_unstale":"2024-02-14T17:41:31.022983+0000","last_undegraded":"2024-02-14T17:41:31.022983+0000","last_fullsized":"2024-02-14T17:41:31.022983+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:07.393174+0000","last_clean_scrub_stamp":"2024-02-14T17:41:07.393174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T21:40:20.606034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025024+0000","last_change":"2024-02-14T17:41:02.376503+0000","last_active":"2024-02-14T17:41:31.025024+0000","last_peered":"2024-02-14T17:41:31.025024+0000","last_clean":"2024-02-14T17:41:31.025024+0000","last_became_active":"2024-02-14T17:41:02.376195+0000","last_became_peered":"2024-02-14T17:41:02.376195+0000","last_unstale":"2024-02-14T17:41:31.025024+0000","last_undegraded":"2024-02-14T17:41:31.025024+0000","last_fullsized":"2024-02-14T17:41:31.025024+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:01.350109+0000","last_clean_scrub_stamp":"2024-02-14T17:41:01.350109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T19:04:35.248705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.024894+0000","last_change":"2024-02-14T17:41:04.384945+0000","last_active":"2024-02-14T17:41:31.024894+0000","last_peered":"2024-02-14T17:41:31.024894+0000","last_clean":"2024-02-14T17:41:31.024894+0000","last_became_active":"2024-02-14T17:41:04.384213+0000","last_became_peered":"2024-02-14T17:41:04.384213+0000","last_unstale":"2024-02-14T17:41:31.024894+0000","last_undegraded":"2024-02-14T17:41:31.024894+0000","last_fullsized":"2024-02-14T17:41:31.024894+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:03.366203+0000","last_clean_scrub_stamp":"2024-02-14T17:41:03.366203+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T02:53:48.390856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-14T17:41:31.025290+0000","last_change":"2024-02-14T17:41:06.398471+0000","last_active":"2024-02-14T17:41:31.025290+0000","last_peered":"2024-02-14T17:41:31.025290+0000","last_clean":"2024-02-14T17:41:31.025290+0000","last_became_active":"2024-02-14T17:41:06.397810+0000","last_became_peered":"2024-02-14T17:41:06.397810+0000","last_unstale":"2024-02-14T17:41:31.025290+0000","last_undegraded":"2024-02-14T17:41:31.025290+0000","last_fullsized":"2024-02-14T17:41:31.025290+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:41:05.383463+0000","last_clean_scrub_stamp":"2024-02-14T17:41:05.383463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-16T01:36:46.024003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411344,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.374},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.39700000000000002}]},{"osd":1,"last update":"Wed Feb 14 17:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.496}]}]},{"osd":0,"up_from":8,"seq":34359738395,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 14 17:41:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.749}]},{"osd":2,"last update":"Wed Feb 14 17:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.73099999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.57599999999999996}]},{"osd":2,"last update":"Wed Feb 14 17:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.375}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-14T17:42:01.063 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-14T17:42:01.064 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-14T17:42:01.064 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-14T17:42:01.064 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- ceph health --format=json 2024-02-14T17:42:01.284 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:01 smithi027 ceph-mon[25828]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:42:01.414 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:01 smithi096 ceph-mon[27796]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:42:01.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:01 smithi040 ceph-mon[27571]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:42:01.703 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:42:02.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:02 smithi027 ceph-mon[25828]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:02.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:02 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:02.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:02 smithi040 ceph-mon[27571]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:02.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:02 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:02.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:02 smithi096 ceph-mon[27796]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:02.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:02 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:03.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:03 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:03.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:03 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:03.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:03 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:03.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:03 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:03.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:03 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:03.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:03 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:03.728 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:03 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:03.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:03 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:03.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:03 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:04.534 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:04 smithi027 ceph-mon[25828]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:04.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:04 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:04.728 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:04 smithi040 ceph-mon[27571]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:04.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:04 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:04.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:04 smithi096 ceph-mon[27796]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:04.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:04 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:04.844 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-02-14T17:42:04.844 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-14T17:42:05.155 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:04 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[48745]: ts=2024-02-14T17:42:04.973Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003719916s 2024-02-14T17:42:05.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:05 smithi040 ceph-mon[27571]: from='client.? 172.21.15.27:0/4092536185' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-14T17:42:05.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:05 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:05 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:05 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:05 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:05 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:05 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:05 smithi027 ceph-mon[25828]: from='client.? 172.21.15.27:0/4092536185' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-14T17:42:05.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:05 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:05 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:05 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:05 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:05 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:05 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.602 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-14T17:42:05.603 INFO:tasks.cephadm:Setup complete, yielding 2024-02-14T17:42:05.603 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-14T17:42:05.612 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-02-14T17:42:05.613 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-02-14T17:42:05.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:05 smithi096 ceph-mon[27796]: from='client.? 172.21.15.27:0/4092536185' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-14T17:42:05.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:05 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:05 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:05 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:05 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:05 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:05 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:05.863 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:42:06.678 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:06 smithi027 ceph-mon[25828]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:06.679 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:06 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: ::ffff:172.21.15.40 - - [14/Feb/2024:17:42:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-14T17:42:06.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:06 smithi040 ceph-mon[27571]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:06.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:06 smithi096 ceph-mon[27796]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:07.691 INFO:teuthology.orchestra.run.smithi027.stdout:167 167 2024-02-14T17:42:08.361 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- bash -c 'ceph orch status' 2024-02-14T17:42:08.531 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:08 smithi040 ceph-mon[27571]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:08.619 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:42:08.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:08 smithi027 ceph-mon[25828]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:08.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:08 smithi096 ceph-mon[27796]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:10.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:10 smithi027 ceph-mon[25828]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:10.552 INFO:teuthology.orchestra.run.smithi027.stdout:Backend: cephadm 2024-02-14T17:42:10.552 INFO:teuthology.orchestra.run.smithi027.stdout:Available: Yes 2024-02-14T17:42:10.552 INFO:teuthology.orchestra.run.smithi027.stdout:Paused: No 2024-02-14T17:42:10.728 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:10 smithi040 ceph-mon[27571]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:10.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:10 smithi096 ceph-mon[27796]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:11.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-14T17:42:11.415 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 systemd[1]: Starting Ceph grafana.a for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:11.513 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:42:11.674 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 podman[38728]: 2024-02-14 17:42:11.316383056 +0000 UTC m=+0.016446151 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-02-14T17:42:11.674 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 podman[38728]: 2024-02-14 17:42:11.427455297 +0000 UTC m=+0.127518420 container create c8a0478db7db24db66774648fbf1b6cb271854285ec8c691e39e80d64cfc264c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.openshift.tags=base rhel8, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, release=239.1651231664, vendor=Red Hat, Inc., name=ubi8, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, maintainer=Paul Cuzner ) 2024-02-14T17:42:11.674 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 podman[38728]: 2024-02-14 17:42:11.672624299 +0000 UTC m=+0.372687384 container init c8a0478db7db24db66774648fbf1b6cb271854285ec8c691e39e80d64cfc264c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5) 2024-02-14T17:42:11.924 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 podman[38728]: 2024-02-14 17:42:11.67804594 +0000 UTC m=+0.378109018 container start c8a0478db7db24db66774648fbf1b6cb271854285ec8c691e39e80d64cfc264c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a, build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, name=ubi8, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, description=Ceph Grafana Container, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., architecture=x86_64, release=239.1651231664, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , com.redhat.component=ubi8-container) 2024-02-14T17:42:11.924 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 bash[38728]: c8a0478db7db24db66774648fbf1b6cb271854285ec8c691e39e80d64cfc264c 2024-02-14T17:42:11.924 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 systemd[1]: Started Ceph grafana.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:42:11.924 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.837998834Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-02-14T17:42:11.925 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.838190533Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-02-14T17:42:11.925 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.838210163Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-02-14T17:42:11.925 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.83821847Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-02-14T17:42:11.925 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.838224833Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-14T17:42:11.925 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.838231032Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-14T17:42:11.925 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.838237495Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-14T17:42:11.925 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.838243624Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-14T17:42:11.925 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.838249685Z level=info msg="Path Home" path=/usr/share/grafana 2024-02-14T17:42:11.925 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.838255414Z level=info msg="Path Data" path=/var/lib/grafana 2024-02-14T17:42:11.926 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.838261054Z level=info msg="Path Logs" path=/var/log/grafana 2024-02-14T17:42:11.926 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.83826659Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-02-14T17:42:11.926 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.838272445Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-02-14T17:42:11.926 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=settings t=2024-02-14T17:42:11.838278887Z level=info msg="App mode production" 2024-02-14T17:42:11.926 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=sqlstore t=2024-02-14T17:42:11.838321963Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-02-14T17:42:11.926 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=sqlstore t=2024-02-14T17:42:11.838338454Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-14T17:42:11.926 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.839353531Z level=info msg="Starting DB migrations" 2024-02-14T17:42:11.926 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.839432156Z level=info msg="Executing migration" id="create migration_log table" 2024-02-14T17:42:11.926 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.840015524Z level=info msg="Executing migration" id="create user table" 2024-02-14T17:42:11.926 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.840447001Z level=info msg="Executing migration" id="add unique index user.login" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.843281786Z level=info msg="Executing migration" id="add unique index user.email" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.84599753Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.848592479Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.851085575Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.853619181Z level=info msg="Executing migration" id="create user table v2" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.856158744Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.858787627Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.859617625Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.860371844Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.861216867Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.862055586Z level=info msg="Executing migration" id="Update user table charset" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.862878327Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.863353194Z level=info msg="Executing migration" id="Add missing user data" 2024-02-14T17:42:11.927 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.86418093Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-02-14T17:42:11.928 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.864951594Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-02-14T17:42:11.928 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.865748182Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-02-14T17:42:11.928 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.866543057Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-02-14T17:42:11.928 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.86771721Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-02-14T17:42:11.928 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.868570471Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-02-14T17:42:11.928 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.869330493Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-02-14T17:42:11.928 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.870212412Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-02-14T17:42:11.929 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.870969072Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-02-14T17:42:11.929 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.871778575Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-02-14T17:42:11.929 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.872541341Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-02-14T17:42:11.929 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.873433776Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-02-14T17:42:11.929 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.874392168Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-02-14T17:42:11.929 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.875115771Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-02-14T17:42:11.929 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.875979971Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-14T17:42:11.929 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.876739426Z level=info msg="Executing migration" id="create temp_user v2" 2024-02-14T17:42:11.929 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.877608964Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-02-14T17:42:11.929 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.878423514Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.87886931Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.879796111Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.880604Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.881410309Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.882259432Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.883156841Z level=info msg="Executing migration" id="create star table" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.884000968Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.88493299Z level=info msg="Executing migration" id="create org table v1" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.885748611Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.886611721Z level=info msg="Executing migration" id="create org_user table v1" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.887449126Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.890074779Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.8928529Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-02-14T17:42:11.930 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.89567563Z level=info msg="Executing migration" id="Update org table charset" 2024-02-14T17:42:11.931 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.898450394Z level=info msg="Executing migration" id="Update org_user table charset" 2024-02-14T17:42:11.931 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.901218729Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-02-14T17:42:11.931 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.903853846Z level=info msg="Executing migration" id="create dashboard table" 2024-02-14T17:42:11.931 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.906628166Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-02-14T17:42:11.931 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.909472015Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-02-14T17:42:11.931 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.912259689Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-02-14T17:42:11.931 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.914788921Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-02-14T17:42:11.931 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.917637207Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-14T17:42:11.931 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.920381683Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-02-14T17:42:11.931 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.923117015Z level=info msg="Executing migration" id="create dashboard v2" 2024-02-14T17:42:11.931 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.92572763Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-02-14T17:42:12.175 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.928453202Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-02-14T17:42:12.175 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.931215309Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-02-14T17:42:12.175 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.933993366Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-02-14T17:42:12.176 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.936744144Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-02-14T17:42:12.176 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.939524515Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-02-14T17:42:12.176 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.941977466Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-02-14T17:42:12.176 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.944631853Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-02-14T17:42:12.176 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.947237105Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-02-14T17:42:12.176 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.949869848Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-02-14T17:42:12.176 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.952274182Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-02-14T17:42:12.176 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.955165026Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-02-14T17:42:12.176 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.957508153Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-02-14T17:42:12.176 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.960073932Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-02-14T17:42:12.176 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.962517062Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-02-14T17:42:12.176 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.965007368Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-02-14T17:42:12.176 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.967668062Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-02-14T17:42:12.177 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.970456752Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-02-14T17:42:12.177 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.97293612Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-02-14T17:42:12.177 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.973301087Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-02-14T17:42:12.177 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.974144696Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-02-14T17:42:12.177 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.975227442Z level=info msg="Executing migration" id="Update dashboard title length" 2024-02-14T17:42:12.177 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.975794149Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-14T17:42:12.178 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.976708138Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-02-14T17:42:12.178 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.977445418Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-14T17:42:12.178 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.97838334Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-02-14T17:42:12.178 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.979211071Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-14T17:42:12.178 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.980231626Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-14T17:42:12.178 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.980863412Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-02-14T17:42:12.178 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.981671328Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-02-14T17:42:12.178 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.982708521Z level=info msg="Executing migration" id="Add check_sum column" 2024-02-14T17:42:12.178 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.98334377Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-02-14T17:42:12.178 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.984170398Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-02-14T17:42:12.178 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.98509409Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-02-14T17:42:12.178 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.9857513Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-02-14T17:42:12.178 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.986817379Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.987627827Z level=info msg="Executing migration" id="create data_source table" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.988352714Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.988791898Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.989235532Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.990125011Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.991081333Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.992393903Z level=info msg="Executing migration" id="create data_source table v2" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.993384577Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.994114526Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.995010618Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.995997803Z level=info msg="Executing migration" id="Add column with_credentials" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.996725198Z level=info msg="Executing migration" id="Add secure json data column" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.99769427Z level=info msg="Executing migration" id="Update data_source table charset" 2024-02-14T17:42:12.179 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.998463592Z level=info msg="Executing migration" id="Update initial version to 1" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:11 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:11.999316143Z level=info msg="Executing migration" id="Add read_only data column" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.000170361Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.001166546Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.003947382Z level=info msg="Executing migration" id="Add uid column" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.00677206Z level=info msg="Executing migration" id="Update uid value" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.009441525Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.012259002Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.01515669Z level=info msg="Executing migration" id="create api_key table" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.018054383Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.020726897Z level=info msg="Executing migration" id="add index api_key.key" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.023647723Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.026614783Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.029231211Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-02-14T17:42:12.180 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.032097951Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.03475364Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.037740209Z level=info msg="Executing migration" id="create api_key table v2" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.038340625Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.041179093Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.04416742Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.046986807Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.047330745Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.047661952Z level=info msg="Executing migration" id="Update api_key table charset" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.047923025Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.048510868Z level=info msg="Executing migration" id="Add service account foreign key" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.050399198Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.052837162Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.053425723Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-02-14T17:42:12.181 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.056195383Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-02-14T17:42:12.183 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.058682832Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-02-14T17:42:12.183 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.061327212Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-02-14T17:42:12.183 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.061754014Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-02-14T17:42:12.183 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.064621585Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-14T17:42:12.183 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.067323583Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-14T17:42:12.183 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.070238324Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-02-14T17:42:12.183 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.070701683Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-02-14T17:42:12.183 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.07361257Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-02-14T17:42:12.183 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.074691012Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-02-14T17:42:12.183 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.075891093Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-14T17:42:12.183 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.078532419Z level=info msg="Executing migration" id="create quota table v1" 2024-02-14T17:42:12.183 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.078950991Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-14T17:42:12.184 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.079890021Z level=info msg="Executing migration" id="Update quota table charset" 2024-02-14T17:42:12.184 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.080604637Z level=info msg="Executing migration" id="create plugin_setting table" 2024-02-14T17:42:12.184 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.081688205Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-14T17:42:12.184 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.082176951Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-02-14T17:42:12.184 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.083011265Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-02-14T17:42:12.184 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.083855937Z level=info msg="Executing migration" id="create session table" 2024-02-14T17:42:12.184 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.084874982Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-02-14T17:42:12.184 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.085221812Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-02-14T17:42:12.184 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.086066204Z level=info msg="Executing migration" id="create playlist table v2" 2024-02-14T17:42:12.184 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.086881179Z level=info msg="Executing migration" id="create playlist item table v2" 2024-02-14T17:42:12.184 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.087734995Z level=info msg="Executing migration" id="Update playlist table charset" 2024-02-14T17:42:12.185 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.088051119Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-02-14T17:42:12.185 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.08909216Z level=info msg="Executing migration" id="drop preferences table v2" 2024-02-14T17:42:12.185 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.089781921Z level=info msg="Executing migration" id="drop preferences table v3" 2024-02-14T17:42:12.185 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.090714805Z level=info msg="Executing migration" id="create preferences table v3" 2024-02-14T17:42:12.185 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.091260479Z level=info msg="Executing migration" id="Update preferences table charset" 2024-02-14T17:42:12.185 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.091574797Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-02-14T17:42:12.185 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.092456931Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-02-14T17:42:12.185 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.093446606Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-02-14T17:42:12.185 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.09418732Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-02-14T17:42:12.185 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.094830378Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.095855541Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.096523591Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.097607888Z level=info msg="Executing migration" id="create alert table v1" 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.098070202Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.098547683Z level=info msg="Executing migration" id="add index alert state" 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.099363778Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.099829409Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.100248805Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.100691429Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.101119236Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.102534453Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.102921754Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.103366099Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-02-14T17:42:12.186 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.103670369Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.10402467Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.104424064Z level=info msg="Executing migration" id="Add column is_default" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.105023682Z level=info msg="Executing migration" id="Add column frequency" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.105657555Z level=info msg="Executing migration" id="Add column send_reminder" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.106310735Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.106949646Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.107403154Z level=info msg="Executing migration" id="Update alert table charset" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.107680327Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.107956407Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.108363038Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.108802444Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.109212694Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.10960826Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.110036784Z level=info msg="Executing migration" id="Add for to alert table" 2024-02-14T17:42:12.187 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.110680071Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.111322105Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.11165892Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.11210425Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.112529905Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.113181472Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.11348941Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.113900317Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.11434055Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.114660391Z level=info msg="Executing migration" id="create annotation table v5" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.115094763Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.115919306Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.116351592Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.116760474Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-02-14T17:42:12.188 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.117227048Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.11806926Z level=info msg="Executing migration" id="Update annotation table charset" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.118879607Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.11953801Z level=info msg="Executing migration" id="Drop category_id index" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.119940288Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.122635562Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.125091497Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.127464556Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.127877207Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.129504623Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.130380617Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.13137141Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.132102202Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.132950093Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.13388084Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-02-14T17:42:12.189 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.134674295Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-02-14T17:42:12.190 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.135460074Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-02-14T17:42:12.190 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.136248875Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-02-14T17:42:12.190 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.137115109Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-02-14T17:42:12.190 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.137959225Z level=info msg="Executing migration" id="Add epoch_end column" 2024-02-14T17:42:12.190 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.139051763Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-02-14T17:42:12.190 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.139855726Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-02-14T17:42:12.190 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.140291666Z level=info msg="Executing migration" id="Move region to single row" 2024-02-14T17:42:12.190 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.141145419Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-02-14T17:42:12.190 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.141957738Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-14T17:42:12.190 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.142740799Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-14T17:42:12.190 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.143173925Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-14T17:42:12.190 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.144064485Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-14T17:42:12.190 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.144888204Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-02-14T17:42:12.191 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.145719388Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-02-14T17:42:12.191 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.146034105Z level=info msg="Executing migration" id="create test_data table" 2024-02-14T17:42:12.191 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.146880369Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-02-14T17:42:12.191 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.147723896Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-02-14T17:42:12.191 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.148619032Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-14T17:42:12.191 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.149174345Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-02-14T17:42:12.191 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.150035133Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-02-14T17:42:12.191 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.150842369Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-02-14T17:42:12.191 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.151724566Z level=info msg="Executing migration" id="create team table" 2024-02-14T17:42:12.191 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.15220184Z level=info msg="Executing migration" id="add index team.org_id" 2024-02-14T17:42:12.191 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.153078061Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-02-14T17:42:12.191 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.153957173Z level=info msg="Executing migration" id="create team member table" 2024-02-14T17:42:12.191 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.154915492Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-02-14T17:42:12.192 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.155549124Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-02-14T17:42:12.192 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.156541123Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-02-14T17:42:12.192 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.159123943Z level=info msg="Executing migration" id="Add column email to team table" 2024-02-14T17:42:12.192 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.161948355Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-02-14T17:42:12.192 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.162791876Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-02-14T17:42:12.192 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.165760291Z level=info msg="Executing migration" id="create dashboard acl table" 2024-02-14T17:42:12.192 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.168359465Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-02-14T17:42:12.192 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.170981063Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-14T17:42:12.192 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.171444442Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-14T17:42:12.192 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.174062362Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-02-14T17:42:12.192 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.176813182Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-02-14T17:42:12.192 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.179618688Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-02-14T17:42:12.192 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.182443711Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-02-14T17:42:12.426 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-mon[27571]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:12.426 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-mon[27571]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:12.427 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:12.427 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:12.427 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:12.427 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:12.428 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.185140582Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-02-14T17:42:12.428 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.18818703Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-02-14T17:42:12.428 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.190731448Z level=info msg="Executing migration" id="create tag table" 2024-02-14T17:42:12.428 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.193510925Z level=info msg="Executing migration" id="add index tag.key_value" 2024-02-14T17:42:12.428 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.19591319Z level=info msg="Executing migration" id="create login attempt table" 2024-02-14T17:42:12.428 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.198744454Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-02-14T17:42:12.428 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.20126002Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-02-14T17:42:12.428 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.204066725Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-14T17:42:12.428 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.206557058Z level=info msg="Executing migration" id="create login_attempt v2" 2024-02-14T17:42:12.428 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.209301896Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.212044002Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.214598518Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.217075763Z level=info msg="Executing migration" id="create user auth table" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.219696363Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.221901927Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.224581237Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.227125257Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.229649568Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.232238713Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.235091909Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.237527918Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.240009577Z level=info msg="Executing migration" id="create server_lock table" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.242625835Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-02-14T17:42:12.429 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.245300357Z level=info msg="Executing migration" id="create user auth token table" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.245944609Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.246791705Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.247636376Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.248561079Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.249400677Z level=info msg="Executing migration" id="create cache_data table" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.250316652Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.251117132Z level=info msg="Executing migration" id="create short_url table v1" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.251890253Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.252897855Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.253820922Z level=info msg="Executing migration" id="delete alert_definition table" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.254386558Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.255220028Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.256022721Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-02-14T17:42:12.430 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.256866989Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.257839118Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.258469622Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.259253248Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.259689187Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.260136733Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.26098737Z level=info msg="Executing migration" id="drop alert_definition table" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.262020765Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.262710028Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.263686776Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.26438059Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.265388623Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.26622149Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.266979014Z level=info msg="Executing migration" id="create alert_instance table" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.267947413Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-14T17:42:12.431 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.268657404Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.269580673Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.270417711Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.271289489Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.272158751Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.275683013Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.279066161Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.281644322Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.284446684Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.287308611Z level=info msg="Executing migration" id="create alert_rule table" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.29007052Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.292740523Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.295720866Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.29845365Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-02-14T17:42:12.432 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.301302202Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-02-14T17:42:12.433 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.303875482Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-02-14T17:42:12.433 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.306634278Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-02-14T17:42:12.433 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.309419292Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-02-14T17:42:12.433 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.312236751Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-14T17:42:12.433 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.315108941Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-02-14T17:42:12.433 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.317716579Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-02-14T17:42:12.434 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.320522509Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-14T17:42:12.434 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.32337331Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-02-14T17:42:12.434 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.326050464Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-02-14T17:42:12.434 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.327244597Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-02-14T17:42:12.434 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.328101004Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-02-14T17:42:12.434 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.328887075Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-14T17:42:12.434 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.329687512Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-14T17:42:12.434 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.330253965Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.330631606Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.331566854Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.332455814Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.333357253Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.334236818Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.335122443Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.335550349Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.33654452Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.33742143Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.338342469Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.33923936Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.340085178Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.340937909Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.341828627Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-02-14T17:42:12.435 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.342280584Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.343170841Z level=info msg="Executing migration" id="create provenance_type table" 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.344048434Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.346851517Z level=info msg="Executing migration" id="create alert_image table" 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.347314893Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.350110021Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.352952028Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.353405131Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.355659737Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.355757876Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.356075538Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.35844019Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.361247228Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.361395351Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-02-14T17:42:12.436 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.361540038Z level=info msg="alerts found to migrate" alerts=0 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.364027147Z level=info msg="Executing migration" id="create library_element table v1" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.36450672Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.367334743Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.370156348Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.372922014Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.373427841Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.376302854Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.376717947Z level=info msg="Executing migration" id="create data_keys table" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.379224684Z level=info msg="Executing migration" id="create secrets table" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.381986889Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.386377757Z level=info msg="Executing migration" id="add name column into data_keys" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.389107497Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.391664017Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.396098903Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.400423814Z level=info msg="Executing migration" id="create kv_store table v1" 2024-02-14T17:42:12.437 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.403104072Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.405794804Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.40844391Z level=info msg="Executing migration" id="create permission table" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.408928578Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.41195944Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.414029267Z level=info msg="Executing migration" id="create role table" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.416822995Z level=info msg="Executing migration" id="add column display_name" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.41783927Z level=info msg="Executing migration" id="add column group_name" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.420470372Z level=info msg="Executing migration" id="add index role.org_id" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.422844515Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.425497503Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.426006088Z level=info msg="Executing migration" id="create team role table" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.426840022Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.427659639Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.428487114Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-02-14T17:42:12.438 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.42897298Z level=info msg="Executing migration" id="create user role table" 2024-02-14T17:42:12.729 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.429771543Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.430671403Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.431429123Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.431878788Z level=info msg="Executing migration" id="create builtin role table" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.432667735Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.433515696Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.434348538Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.435356396Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.435883002Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.436655324Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.437248072Z level=info msg="Executing migration" id="add unique index role.uid" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.437790518Z level=info msg="Executing migration" id="create seed assignment table" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.438205465Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.438647476Z level=info msg="Executing migration" id="add column hidden to role table" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.439696206Z level=info msg="Executing migration" id="create query_history table v1" 2024-02-14T17:42:12.730 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.44011886Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.440569889Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.440836552Z level=info msg="Executing migration" id="teams permissions migration" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.441287633Z level=info msg="Executing migration" id="dashboard permissions" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.441748189Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.442140468Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.44245036Z level=info msg="Executing migration" id="alerting notification permissions" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.442780232Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.443182405Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.443616846Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.444592527Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.444895484Z level=info msg="Executing migration" id="create correlation table v1" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.445368069Z level=info msg="Executing migration" id="add index correlations.uid" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.445800516Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.446232296Z level=info msg="Executing migration" id="add correlation config column" 2024-02-14T17:42:12.731 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.447217639Z level=info msg="Executing migration" id="create entity_events table" 2024-02-14T17:42:12.732 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.44764267Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-02-14T17:42:12.732 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.448075607Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-14T17:42:12.732 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.448185613Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-14T17:42:12.732 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.448489149Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-14T17:42:12.732 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.448593213Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-14T17:42:12.732 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.448869008Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.449299041Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.44974788Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.450605127Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.45108355Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.451856855Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.452777283Z level=info msg="Executing migration" id="Drop public config table" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.453664374Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.454298814Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.455164182Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.456021706Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.456886904Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.461341491Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.46237855Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-02-14T17:42:12.733 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.463430467Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.464279447Z level=info msg="Executing migration" id="add share column" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.465293119Z level=info msg="Executing migration" id="create default alerting folders" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.466179118Z level=info msg="Executing migration" id="create file table" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.467076385Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.468004058Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.46860299Z level=info msg="Executing migration" id="create file_meta table" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.46904401Z level=info msg="Executing migration" id="file table idx: path key" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.471931761Z level=info msg="Executing migration" id="set path collation in file table" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.47460577Z level=info msg="Executing migration" id="managed permissions migration" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.475009222Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.477831079Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.480542479Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.483386945Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.483730051Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-02-14T17:42:12.734 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.48648864Z level=info msg="Executing migration" id="update group index for alert rules" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.489154849Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.491956794Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.492352511Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.495147695Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.497829665Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.500530732Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.510551788Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.513366536Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.516150434Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.519666759Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.520022691Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.522775967Z level=info msg="Executing migration" id="create folder table" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.525399105Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.527941556Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-02-14T17:42:12.735 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.528432893Z level=info msg="Executing migration" id="Update folder title length" 2024-02-14T17:42:12.736 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.531034136Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-02-14T17:42:12.736 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=migrator t=2024-02-14T17:42:12.531510387Z level=info msg="migrations completed" performed=485 skipped=0 duration=692.100141ms 2024-02-14T17:42:12.736 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=sqlstore t=2024-02-14T17:42:12.531944889Z level=info msg="Created default organization" 2024-02-14T17:42:12.736 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=plugin.loader t=2024-02-14T17:42:12.590703941Z level=info msg="Plugin registered" pluginID=input 2024-02-14T17:42:12.736 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=plugin.loader t=2024-02-14T17:42:12.620102858Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-02-14T17:42:12.736 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=plugin.loader t=2024-02-14T17:42:12.62012783Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-02-14T17:42:12.736 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=secrets t=2024-02-14T17:42:12.620164086Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-02-14T17:42:12.736 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=query_data t=2024-02-14T17:42:12.622350645Z level=info msg="Query Service initialization" 2024-02-14T17:42:12.736 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:12 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=live.push_http t=2024-02-14T17:42:12.624923251Z level=info msg="Live Push Gateway initialization" 2024-02-14T17:42:12.736 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:12 smithi027 ceph-mon[25828]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:12.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:12 smithi027 ceph-mon[25828]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:12.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:12 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:12.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:12 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:12.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:12 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:12.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:12 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:12.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:12 smithi096 ceph-mon[27796]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:12.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:12 smithi096 ceph-mon[27796]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:12.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:12 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:12.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:12 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:12.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:12 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:12.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:12 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:13.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:13 smithi027 ceph-mon[25828]: Deploying daemon node-exporter.a on smithi027 2024-02-14T17:42:13.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-mon[27571]: Deploying daemon node-exporter.a on smithi027 2024-02-14T17:42:13.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=infra.usagestats.collector t=2024-02-14T17:42:13.303077164Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-02-14T17:42:13.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=provisioning.datasources t=2024-02-14T17:42:13.304188536Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-14T17:42:13.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=provisioning.datasources t=2024-02-14T17:42:13.315328134Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-02-14T17:42:13.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=provisioning.alerting t=2024-02-14T17:42:13.328169936Z level=info msg="starting to provision alerting" 2024-02-14T17:42:13.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=provisioning.alerting t=2024-02-14T17:42:13.328189945Z level=info msg="finished to provision alerting" 2024-02-14T17:42:13.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=grafanaStorageLogger t=2024-02-14T17:42:13.328427505Z level=info msg="storage starting" 2024-02-14T17:42:13.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=ngalert.state.manager t=2024-02-14T17:42:13.328803857Z level=info msg="Warming state cache for startup" 2024-02-14T17:42:13.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=ngalert.state.manager t=2024-02-14T17:42:13.329057414Z level=info msg="State cache has been initialized" states=0 duration=252.177µs 2024-02-14T17:42:13.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=ngalert.multiorg.alertmanager t=2024-02-14T17:42:13.329291355Z level=info msg="starting MultiOrg Alertmanager" 2024-02-14T17:42:13.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=ticker t=2024-02-14T17:42:13.329323134Z level=info msg=starting first_tick=2024-02-14T17:42:20Z 2024-02-14T17:42:13.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=http.server t=2024-02-14T17:42:13.335061397Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-02-14T17:42:13.480 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=sqlstore.transactions t=2024-02-14T17:42:13.347715358Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-14T17:42:13.480 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=sqlstore.transactions t=2024-02-14T17:42:13.348112607Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-14T17:42:13.480 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=sqlstore.transactions t=2024-02-14T17:42:13.358148571Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-02-14T17:42:13.480 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:42:13 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a[38742]: logger=sqlstore.transactions t=2024-02-14T17:42:13.3792424Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-14T17:42:13.657 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:42:13.657 INFO:teuthology.orchestra.run.smithi027.stdout:agent.smithi027 smithi027 running 8s ago 3m - - 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:agent.smithi040 smithi040 running 8s ago 3m - - 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:agent.smithi096 smithi096 running 11s ago 3m - - 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.a smithi027 *:9093,9094 running (19s) 8s ago 18s 16.5M - 0.25.0 c8568f914cd2 c74a7ffc7aa9 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.a smithi040 *:3000 starting - - - - 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.x smithi040 *:8443,8765 running (2m) 8s ago 2m 448M - 19.0.0-1430-g8727ce93 170846cf6432 9e3b48ef91d7 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.y smithi027 *:9283,8765 running (4m) 8s ago 4m 502M - 19.0.0-1430-g8727ce93 170846cf6432 87bd103ac732 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.z smithi096 *:8443,8765 running (2m) 11s ago 2m 449M - 19.0.0-1430-g8727ce93 170846cf6432 9ec91534454f 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:mon.a smithi027 running (4m) 8s ago 5m 46.0M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6714e9fe1fa6 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:mon.b smithi040 running (3m) 8s ago 3m 39.0M 2048M 19.0.0-1430-g8727ce93 170846cf6432 544f97a89dd6 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:mon.c smithi096 running (3m) 11s ago 3m 40.4M 2048M 19.0.0-1430-g8727ce93 170846cf6432 ef7f0c7cea0a 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (2m) 8s ago 2m 58.4M 4096M 19.0.0-1430-g8727ce93 170846cf6432 5bf0e4876baf 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi040 running (117s) 8s ago 117s 62.7M 4096M 19.0.0-1430-g8727ce93 170846cf6432 9b75dab64136 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi096 running (92s) 11s ago 91s 53.0M 4096M 19.0.0-1430-g8727ce93 170846cf6432 c53d1a6d5381 2024-02-14T17:42:13.658 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.a smithi040 *:9095 running (51s) 8s ago 51s 29.8M - 2.43.0 a07b618ecd1d 1f71374f1913 2024-02-14T17:42:13.659 INFO:teuthology.orchestra.run.smithi027.stdout:rgw.foo.a smithi027 *:80 running (73s) 8s ago 73s 77.0M - 19.0.0-1430-g8727ce93 170846cf6432 77b6b21cb6a8 2024-02-14T17:42:13.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:13 smithi096 ceph-mon[27796]: Deploying daemon node-exporter.a on smithi027 2024-02-14T17:42:13.973 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:13 smithi027 systemd[1]: Starting Ceph node-exporter.a for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:14.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-14T17:42:14.544 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:14 smithi027 ceph-mon[25828]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:14.585 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:14 smithi040 ceph-mon[27571]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:14.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:14 smithi096 ceph-mon[27796]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:15.004 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:42:15.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:15 smithi027 ceph-mon[25828]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:15.370 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:15 smithi027 bash[50846]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-14T17:42:15.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:15 smithi040 ceph-mon[27571]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:15.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:15 smithi096 ceph-mon[27796]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:15.890 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:15 smithi027 bash[50846]: Getting image source signatures 2024-02-14T17:42:15.890 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:15 smithi027 bash[50846]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-14T17:42:15.890 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:15 smithi027 bash[50846]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-14T17:42:15.890 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:15 smithi027 bash[50846]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-14T17:42:16.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:16 smithi040 ceph-mon[27571]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:16.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:16 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:16.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:16 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:16.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:16 smithi027 ceph-mon[25828]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:16.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:16 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:16.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:16 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:16.733 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:16 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: ::ffff:172.21.15.40 - - [14/Feb/2024:17:42:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-14T17:42:16.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:16 smithi096 ceph-mon[27796]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:16.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:16 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:16.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:16 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:17.596 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:17 smithi027 bash[50846]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-14T17:42:17.906 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:17 smithi027 bash[50846]: Writing manifest to image destination 2024-02-14T17:42:18.095 INFO:teuthology.orchestra.run.smithi027.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-14T17:42:18.095 INFO:teuthology.orchestra.run.smithi027.stdout:agent 3/3 15s ago 47s * 2024-02-14T17:42:18.095 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager ?:9093,9094 1/1 13s ago 72s smithi027=a;count:1 2024-02-14T17:42:18.095 INFO:teuthology.orchestra.run.smithi027.stdout:grafana ?:3000 0/1 - 69s smithi040=a;count:1 2024-02-14T17:42:18.095 INFO:teuthology.orchestra.run.smithi027.stdout:mgr 3/3 15s ago 2m smithi027=y;smithi040=x;smithi096=z;count:3 2024-02-14T17:42:18.095 INFO:teuthology.orchestra.run.smithi027.stdout:mon 3/3 15s ago 3m smithi027:172.21.15.27=a;smithi040:172.21.15.40=b;smithi096:172.21.15.96=c;count:3 2024-02-14T17:42:18.095 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter ?:9100 0/3 - 75s smithi027=a;smithi040=b;smithi096=c;count:3 2024-02-14T17:42:18.095 INFO:teuthology.orchestra.run.smithi027.stdout:osd 3 15s ago - 2024-02-14T17:42:18.095 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus ?:9095 1/1 12s ago 78s smithi040=a;count:1 2024-02-14T17:42:18.095 INFO:teuthology.orchestra.run.smithi027.stdout:rgw.foo.a ?:80 1/1 13s ago 77s smithi027=foo.a;count:1 2024-02-14T17:42:18.233 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:18 smithi027 podman[50846]: 2024-02-14 17:42:18.008655491 +0000 UTC m=+3.341435227 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-14T17:42:18.233 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:18 smithi027 podman[50846]: 2024-02-14 17:42:18.09854499 +0000 UTC m=+3.431324719 container create fffd927061487782ca1811a5bdacc2bb36677d9a82f7b29375e1ecbfbb2cd247 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:18.670 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:18 smithi040 ceph-mon[27571]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:18.670 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:18 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:18.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:18 smithi027 ceph-mon[25828]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:18.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:18 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:18.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:18 smithi096 ceph-mon[27796]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:18.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:18 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:19.140 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-14T17:42:19.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-mon[25828]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:19.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:19.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:19.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:19.373 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 podman[50846]: 2024-02-14 17:42:19.262749979 +0000 UTC m=+4.595529708 container init fffd927061487782ca1811a5bdacc2bb36677d9a82f7b29375e1ecbfbb2cd247 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:19.373 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 podman[50846]: 2024-02-14 17:42:19.269892936 +0000 UTC m=+4.602672670 container start fffd927061487782ca1811a5bdacc2bb36677d9a82f7b29375e1ecbfbb2cd247 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:19.373 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.273Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-14T17:42:19.373 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.273Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-14T17:42:19.373 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.274Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-14T17:42:19.373 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.274Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-14T17:42:19.373 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.274Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-14T17:42:19.373 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.274Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-14T17:42:19.373 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-14T17:42:19.373 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=arp 2024-02-14T17:42:19.374 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-14T17:42:19.374 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-14T17:42:19.374 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-14T17:42:19.374 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-14T17:42:19.374 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-14T17:42:19.374 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-14T17:42:19.374 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-14T17:42:19.374 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-14T17:42:19.374 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=edac 2024-02-14T17:42:19.374 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-14T17:42:19.374 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-14T17:42:19.374 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-14T17:42:19.375 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-14T17:42:19.375 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-14T17:42:19.375 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-14T17:42:19.375 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-14T17:42:19.375 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-14T17:42:19.375 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-14T17:42:19.375 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-14T17:42:19.375 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-14T17:42:19.375 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-14T17:42:19.375 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-14T17:42:19.375 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-14T17:42:19.375 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-14T17:42:19.375 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-14T17:42:19.376 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=os 2024-02-14T17:42:19.376 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-14T17:42:19.376 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-14T17:42:19.376 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=stat 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=time 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=uname 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-14T17:42:19.377 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-14T17:42:19.378 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a[51072]: ts=2024-02-14T17:42:19.275Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-14T17:42:19.378 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 bash[50846]: fffd927061487782ca1811a5bdacc2bb36677d9a82f7b29375e1ecbfbb2cd247 2024-02-14T17:42:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:42:19.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:19 smithi040 ceph-mon[27571]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:19.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:19 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:19.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:19 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:19.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:19 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:19.699 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:42:19 smithi027 systemd[1]: Started Ceph node-exporter.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:42:19.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:19 smithi096 ceph-mon[27796]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:19.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:19 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:19.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:19 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:19.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:19 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:20.631 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:20 smithi040 ceph-mon[27571]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:20.631 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:20 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:20.632 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:20 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:20.632 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:20 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:20.632 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:20 smithi040 ceph-mon[27571]: Deploying daemon node-exporter.b on smithi040 2024-02-14T17:42:20.632 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:20 smithi040 systemd[1]: Starting Ceph node-exporter.b for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:20 smithi027 ceph-mon[25828]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:20 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:20 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:20 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:20 smithi027 ceph-mon[25828]: Deploying daemon node-exporter.b on smithi040 2024-02-14T17:42:20.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:20 smithi096 ceph-mon[27796]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:20.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:20 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:20.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:20 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:20.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:20 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:20.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:20 smithi096 ceph-mon[27796]: Deploying daemon node-exporter.b on smithi040 2024-02-14T17:42:20.979 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:20 smithi040 bash[39761]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-14T17:42:21.478 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:21 smithi040 bash[39761]: Getting image source signatures 2024-02-14T17:42:21.479 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:21 smithi040 bash[39761]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-14T17:42:21.479 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:21 smithi040 bash[39761]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-14T17:42:21.479 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:21 smithi040 bash[39761]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-14T17:42:22.517 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:22 smithi027 ceph-mon[25828]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:22.517 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:22 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:22.517 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:22 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:22.517 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:22 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:22.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:22 smithi096 ceph-mon[27796]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:22.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:22 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:22.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:22 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:22.546 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:22 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:22.728 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:22 smithi040 bash[39761]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-14T17:42:22.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:22 smithi040 ceph-mon[27571]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:22.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:22 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:22.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:22 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:22.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:22 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:22.768 INFO:teuthology.orchestra.run.smithi027.stdout:HOST ADDR LABELS STATUS 2024-02-14T17:42:22.768 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 172.21.15.27 2024-02-14T17:42:22.768 INFO:teuthology.orchestra.run.smithi027.stdout:smithi040 172.21.15.40 2024-02-14T17:42:22.769 INFO:teuthology.orchestra.run.smithi027.stdout:smithi096 172.21.15.96 2024-02-14T17:42:22.769 INFO:teuthology.orchestra.run.smithi027.stdout:3 hosts in cluster 2024-02-14T17:42:23.164 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:22 smithi040 bash[39761]: Writing manifest to image destination 2024-02-14T17:42:23.457 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 podman[39761]: 2024-02-14 17:42:23.106162215 +0000 UTC m=+2.570420258 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-14T17:42:23.458 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 podman[39761]: 2024-02-14 17:42:23.230012518 +0000 UTC m=+2.694270554 container create 9ef0ae1a5738346c9d1a3451413be3279f76b24b6fbcb317af1fe5cbb7e92007 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:42:23.458 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 podman[39761]: 2024-02-14 17:42:23.456889209 +0000 UTC m=+2.921147246 container init 9ef0ae1a5738346c9d1a3451413be3279f76b24b6fbcb317af1fe5cbb7e92007 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:42:23.500 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-14T17:42:23.729 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 podman[39761]: 2024-02-14 17:42:23.462891558 +0000 UTC m=+2.927149595 container start 9ef0ae1a5738346c9d1a3451413be3279f76b24b6fbcb317af1fe5cbb7e92007 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:42:23.729 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.467Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-14T17:42:23.729 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.467Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.467Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.467Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=arp 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-14T17:42:23.730 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-14T17:42:23.731 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=edac 2024-02-14T17:42:23.731 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-14T17:42:23.731 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-14T17:42:23.731 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-14T17:42:23.731 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-14T17:42:23.731 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-14T17:42:23.731 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-14T17:42:23.731 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=os 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-14T17:42:23.732 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-14T17:42:23.733 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-14T17:42:23.733 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-14T17:42:23.733 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=stat 2024-02-14T17:42:23.733 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-14T17:42:23.733 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-14T17:42:23.733 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-14T17:42:23.733 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=time 2024-02-14T17:42:23.733 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-14T17:42:23.733 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=uname 2024-02-14T17:42:23.733 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-14T17:42:23.733 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-14T17:42:23.733 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.468Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-14T17:42:23.734 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.469Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-14T17:42:23.734 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b[39810]: ts=2024-02-14T17:42:23.469Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-14T17:42:23.734 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 bash[39761]: 9ef0ae1a5738346c9d1a3451413be3279f76b24b6fbcb317af1fe5cbb7e92007 2024-02-14T17:42:23.734 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:42:23 smithi040 systemd[1]: Started Ceph node-exporter.b for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:42:24.096 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:42:24.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:24 smithi027 ceph-mon[25828]: from='client.24404 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:24.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:24 smithi027 ceph-mon[25828]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:24.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:24 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:24.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:24 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:24.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:24 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:24.518 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:24 smithi096 ceph-mon[27796]: from='client.24404 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:24.518 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:24 smithi096 ceph-mon[27796]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:24.518 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:24 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:24.518 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:24 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:24.518 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:24 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:24.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:24 smithi040 ceph-mon[27571]: from='client.24404 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:24.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:24 smithi040 ceph-mon[27571]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:24.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:24 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:24.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:24 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:24.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:24 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:24.795 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:24 smithi096 systemd[1]: Starting Ceph node-exporter.c for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:25.295 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:24 smithi096 bash[39898]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-14T17:42:25.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:25 smithi027 ceph-mon[25828]: Deploying daemon node-exporter.c on smithi096 2024-02-14T17:42:25.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:25 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:25.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:25 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:25.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:25 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:25.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:25 smithi040 ceph-mon[27571]: Deploying daemon node-exporter.c on smithi096 2024-02-14T17:42:25.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:25 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:25.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:25 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:25.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:25 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:25.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:25 smithi096 ceph-mon[27796]: Deploying daemon node-exporter.c on smithi096 2024-02-14T17:42:25.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:25 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:25.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:25 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:25.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:25 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:25.796 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:25 smithi096 bash[39898]: Getting image source signatures 2024-02-14T17:42:25.796 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:25 smithi096 bash[39898]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-14T17:42:25.796 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:25 smithi096 bash[39898]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-14T17:42:25.796 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:25 smithi096 bash[39898]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-14T17:42:26.221 INFO:teuthology.orchestra.run.smithi027.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-14T17:42:26.221 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001C400BGN 372G No 1s ago Has a FileSystem, LVM detected 2024-02-14T17:42:26.221 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme1n1 ssd Linux_311c781b5f37a8107fb3 89.4G Yes 1s ago 2024-02-14T17:42:26.221 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme2n1 ssd Linux_a1926a06b6dc83827cec 89.4G Yes 1s ago 2024-02-14T17:42:26.221 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme3n1 ssd Linux_c9f31ed2f1cabfc0c095 89.4G Yes 1s ago 2024-02-14T17:42:26.221 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme4n1 ssd Linux_f844bf9d30a0971eb28b 89.4G No 1s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-14T17:42:26.221 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W4H3N7 931G No 1s ago Has partitions 2024-02-14T17:42:26.221 INFO:teuthology.orchestra.run.smithi027.stdout:smithi040 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009J400BGN 372G No 7s ago Has a FileSystem, LVM detected 2024-02-14T17:42:26.221 INFO:teuthology.orchestra.run.smithi027.stdout:smithi040 /dev/nvme1n1 ssd Linux_6b252fd59f256fc4ca34 89.4G Yes 7s ago 2024-02-14T17:42:26.221 INFO:teuthology.orchestra.run.smithi027.stdout:smithi040 /dev/nvme2n1 ssd Linux_4a41ee4c958baeeff002 89.4G Yes 7s ago 2024-02-14T17:42:26.221 INFO:teuthology.orchestra.run.smithi027.stdout:smithi040 /dev/nvme3n1 ssd Linux_af1c9db0af8f0d34ae41 89.4G Yes 7s ago 2024-02-14T17:42:26.221 INFO:teuthology.orchestra.run.smithi027.stdout:smithi040 /dev/nvme4n1 ssd Linux_eb321353cc544bde3c85 89.4G No 7s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-14T17:42:26.222 INFO:teuthology.orchestra.run.smithi027.stdout:smithi040 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W4HQ07 931G No 7s ago Has partitions 2024-02-14T17:42:26.222 INFO:teuthology.orchestra.run.smithi027.stdout:smithi096 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NH400BGN 372G No 3s ago Has a FileSystem, LVM detected 2024-02-14T17:42:26.222 INFO:teuthology.orchestra.run.smithi027.stdout:smithi096 /dev/nvme1n1 ssd Linux_efcfc93cf01bf074f5c1 89.4G Yes 3s ago 2024-02-14T17:42:26.222 INFO:teuthology.orchestra.run.smithi027.stdout:smithi096 /dev/nvme2n1 ssd Linux_7945d4a6074f397999b4 89.4G Yes 3s ago 2024-02-14T17:42:26.222 INFO:teuthology.orchestra.run.smithi027.stdout:smithi096 /dev/nvme3n1 ssd Linux_f7819e38379f6a7a6851 89.4G Yes 3s ago 2024-02-14T17:42:26.222 INFO:teuthology.orchestra.run.smithi027.stdout:smithi096 /dev/nvme4n1 ssd Linux_fbdc637b106fd2109170 89.4G No 3s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-14T17:42:26.222 INFO:teuthology.orchestra.run.smithi027.stdout:smithi096 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5QGMP 931G No 3s ago Has partitions 2024-02-14T17:42:26.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:26 smithi027 ceph-mon[25828]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:26.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:26 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: ::ffff:172.21.15.40 - - [14/Feb/2024:17:42:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-14T17:42:26.728 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:26 smithi040 ceph-mon[27571]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:26.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:26 smithi096 ceph-mon[27796]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:27.100 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:26 smithi096 bash[39898]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-14T17:42:27.101 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 bash[39898]: Writing manifest to image destination 2024-02-14T17:42:27.482 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-02-14T17:42:27.723 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-mon[27796]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:27.723 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 podman[39898]: 2024-02-14 17:42:27.354721391 +0000 UTC m=+2.629089339 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-14T17:42:27.723 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 podman[39898]: 2024-02-14 17:42:27.461633437 +0000 UTC m=+2.736001380 container create 05081f4ee8f0ed8570ce7346f1c917917a4bc17573c63bf999392bf7650d39dd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-02-14T17:42:27.723 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 podman[39898]: 2024-02-14 17:42:27.655346458 +0000 UTC m=+2.929714401 container init 05081f4ee8f0ed8570ce7346f1c917917a4bc17573c63bf999392bf7650d39dd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-02-14T17:42:27.723 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 podman[39898]: 2024-02-14 17:42:27.661156371 +0000 UTC m=+2.935524318 container start 05081f4ee8f0ed8570ce7346f1c917917a4bc17573c63bf999392bf7650d39dd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-02-14T17:42:27.723 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.665Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-14T17:42:27.724 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.665Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-14T17:42:27.724 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-14T17:42:27.724 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-14T17:42:27.724 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-14T17:42:27.724 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-14T17:42:27.724 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-14T17:42:27.724 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=arp 2024-02-14T17:42:27.724 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-14T17:42:27.724 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-14T17:42:27.724 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-14T17:42:27.725 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-14T17:42:27.725 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-14T17:42:27.725 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-14T17:42:27.725 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-14T17:42:27.725 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-14T17:42:27.725 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=edac 2024-02-14T17:42:27.725 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-14T17:42:27.725 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-14T17:42:27.725 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-14T17:42:27.725 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-14T17:42:27.725 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-14T17:42:27.725 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=os 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-14T17:42:27.726 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=stat 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=time 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=uname 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.666Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.667Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-14T17:42:27.727 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c[39948]: ts=2024-02-14T17:42:27.667Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-14T17:42:27.728 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 bash[39898]: 05081f4ee8f0ed8570ce7346f1c917917a4bc17573c63bf999392bf7650d39dd 2024-02-14T17:42:27.728 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:27 smithi040 ceph-mon[27571]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:27.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:27 smithi027 ceph-mon[25828]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:28.045 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:42:27 smithi096 systemd[1]: Started Ceph node-exporter.c for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:42:28.193 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:42:28.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:28 smithi027 ceph-mon[25828]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:28.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:28 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:28 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:28 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:28 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:28 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:28 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:28.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:28 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:28.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:28 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:28.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:28 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:28 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:28 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:28 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:28 smithi040 ceph-mon[27571]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:28.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:28 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:28 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:28 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:28 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:28 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:28 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:28.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:28 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:28.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:28 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:28.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:28 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:28 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:28 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.730 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:28 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:28 smithi096 ceph-mon[27796]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:28 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:28 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:28 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:28 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:28 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:28 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:28 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:28 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:28 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:28 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.796 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:28 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:28.797 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:28 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:29.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:29 smithi040 ceph-mon[27571]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:42:29.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:29 smithi040 ceph-mon[27571]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-14T17:42:29.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:29 smithi040 ceph-mon[27571]: Reconfiguring daemon alertmanager.a on smithi027 2024-02-14T17:42:29.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:29 smithi027 ceph-mon[25828]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:42:29.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:29 smithi027 ceph-mon[25828]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-14T17:42:29.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:29 smithi027 ceph-mon[25828]: Reconfiguring daemon alertmanager.a on smithi027 2024-02-14T17:42:29.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:29 smithi096 ceph-mon[27796]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-14T17:42:29.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:29 smithi096 ceph-mon[27796]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-14T17:42:29.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:29 smithi096 ceph-mon[27796]: Reconfiguring daemon alertmanager.a on smithi027 2024-02-14T17:42:30.727 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:30 smithi027 ceph-mon[25828]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:30.728 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:30 smithi040 ceph-mon[27571]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:30.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:30 smithi096 ceph-mon[27796]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:30.889 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: agent 2024-02-14T17:42:30.889 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: agent 2024-02-14T17:42:30.889 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-02-14T17:42:30.889 INFO:teuthology.orchestra.run.smithi027.stdout: host_pattern: '*' 2024-02-14T17:42:30.889 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-02-14T17:42:30.889 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-02-14T17:41:31.079900Z' 2024-02-14T17:42:30.889 INFO:teuthology.orchestra.run.smithi027.stdout: last_refresh: '2024-02-14T17:42:19.101924Z' 2024-02-14T17:42:30.889 INFO:teuthology.orchestra.run.smithi027.stdout: running: 3 2024-02-14T17:42:30.889 INFO:teuthology.orchestra.run.smithi027.stdout: size: 3 2024-02-14T17:42:30.889 INFO:teuthology.orchestra.run.smithi027.stdout:events: 2024-02-14T17:42:30.889 INFO:teuthology.orchestra.run.smithi027.stdout:- 2024-02-14T17:41:32.536186Z service:agent [INFO] "service was created" 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: alertmanager 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: alertmanager 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout: count: 1 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi027=a 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-02-14T17:41:05.533628Z' 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout: last_refresh: '2024-02-14T17:42:28.378910Z' 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout: ports: 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout: - 9093 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout: - 9094 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout: running: 1 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout: size: 1 2024-02-14T17:42:30.890 INFO:teuthology.orchestra.run.smithi027.stdout:events: 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout:- 2024-02-14T17:41:55.134850Z service:alertmanager [INFO] "service was created" 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: grafana 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: grafana 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout: count: 1 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi040=a 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout:spec: 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout: anonymous_access: true 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout: protocol: https 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-02-14T17:41:08.125723Z' 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout: last_refresh: '2024-02-14T17:42:19.102475Z' 2024-02-14T17:42:30.891 INFO:teuthology.orchestra.run.smithi027.stdout: ports: 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout: - 3000 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout: running: 1 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout: size: 1 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout:events: 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout:- 2024-02-14T17:42:11.880848Z service:grafana [INFO] "service was created" 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: mgr 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: mgr 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout: count: 3 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi027=y 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi040=x 2024-02-14T17:42:30.892 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi096=z 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-02-14T17:39:23.959157Z' 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout: last_refresh: '2024-02-14T17:42:19.102157Z' 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout: running: 3 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout: size: 3 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: mon 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: mon 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout: count: 3 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi027:172.21.15.27=a 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi040:172.21.15.40=b 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi096:172.21.15.96=c 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-02-14T17:42:30.893 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-02-14T17:38:54.965778Z' 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout: last_refresh: '2024-02-14T17:42:19.102052Z' 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout: running: 3 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout: size: 3 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: node-exporter 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: node-exporter 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout: count: 3 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi027=a 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi040=b 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi096=c 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-02-14T17:41:02.687998Z' 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout: ports: 2024-02-14T17:42:30.894 INFO:teuthology.orchestra.run.smithi027.stdout: - 9100 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout: running: 1 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout: size: 3 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout:events: 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout:- 2024-02-14T17:42:27.829028Z service:node-exporter [INFO] "service was created" 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: osd 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: osd 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout:unmanaged: true 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout:spec: 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout: filter_logic: AND 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout: objectstore: bluestore 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout: container_image_id: 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:42:30.895 INFO:teuthology.orchestra.run.smithi027.stdout: last_refresh: '2024-02-14T17:42:19.102257Z' 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout: running: 3 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout: size: 3 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: prometheus 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: prometheus 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout: count: 1 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi040=a 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-02-14T17:41:00.012287Z' 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout: last_refresh: '2024-02-14T17:42:19.102354Z' 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout: ports: 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout: - 9095 2024-02-14T17:42:30.896 INFO:teuthology.orchestra.run.smithi027.stdout: running: 1 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout: size: 1 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout:events: 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout:- 2024-02-14T17:42:27.842107Z service:prometheus [INFO] "service was created" 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout:--- 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout:service_type: rgw 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout:service_id: foo.a 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout:service_name: rgw.foo.a 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout:placement: 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout: count: 1 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout: hosts: 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout: - smithi027=foo.a 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout:status: 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout: created: '2024-02-14T17:41:00.672040Z' 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout: last_refresh: '2024-02-14T17:42:28.378769Z' 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout: ports: 2024-02-14T17:42:30.897 INFO:teuthology.orchestra.run.smithi027.stdout: - 80 2024-02-14T17:42:30.898 INFO:teuthology.orchestra.run.smithi027.stdout: running: 1 2024-02-14T17:42:30.898 INFO:teuthology.orchestra.run.smithi027.stdout: size: 1 2024-02-14T17:42:32.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-02-14T17:42:32.189 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:32 smithi096 ceph-mon[27796]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-02-14T17:42:32.189 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:32 smithi096 ceph-mon[27796]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:32.189 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:32 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:32.189 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:32 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:32.189 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:32 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:32.305 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:32 smithi027 ceph-mon[25828]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-02-14T17:42:32.305 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:32 smithi027 ceph-mon[25828]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:32.306 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:32 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:32.306 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:32 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:32.306 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:32 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:32.478 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:32 smithi040 ceph-mon[27571]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-02-14T17:42:32.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:32 smithi040 ceph-mon[27571]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:32.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:32 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:32.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:32 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:32.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:32 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.545 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:33 smithi096 ceph-mon[27796]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.634 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.635 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.635 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.635 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.635 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.635 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.635 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:33 smithi027 ceph-mon[25828]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:33.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:33 smithi040 ceph-mon[27571]: from='mgr.14445 172.21.15.27:0/1141473466' entity='mgr.y' 2024-02-14T17:42:34.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:34 smithi027 ceph-mon[25828]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:34.523 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:42:34.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:34 smithi040 ceph-mon[27571]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:34.733 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:34 smithi027 systemd[1]: Stopping Ceph alertmanager.a for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:34.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:34 smithi096 ceph-mon[27796]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:36.323 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:36 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[48745]: ts=2024-02-14T17:42:36.194Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:42:36.626 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:36 smithi027 podman[52921]: 2024-02-14 17:42:36.3290399 +0000 UTC m=+0.275552070 container died c74a7ffc7aa9e13c044b7202524736aaf9fc6c7d8a3f84d187652cc52fcd0a6b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:36.627 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:36 smithi027 ceph-mon[25828]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:36.627 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:36 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: ::ffff:172.21.15.40 - - [14/Feb/2024:17:42:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-14T17:42:36.729 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:36 smithi040 ceph-mon[27571]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:36.795 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:36 smithi096 ceph-mon[27796]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:42:36.825 INFO:teuthology.orchestra.run.smithi027.stdout:osd 3 17s ago - 2024-02-14T17:42:36.983 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:36 smithi027 podman[52921]: 2024-02-14 17:42:36.724694542 +0000 UTC m=+0.671206722 container cleanup c74a7ffc7aa9e13c044b7202524736aaf9fc6c7d8a3f84d187652cc52fcd0a6b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:36.983 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:36 smithi027 bash[52921]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a 2024-02-14T17:42:37.493 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-14T17:42:37.503 INFO:tasks.cephadm:Teardown begin 2024-02-14T17:42:37.503 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:42:37.538 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:42:37.632 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:42:37.669 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-14T17:42:37.669 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-14T17:42:37.685 DEBUG:teuthology.orchestra.run.smithi040:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-14T17:42:37.702 DEBUG:teuthology.orchestra.run.smithi096:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-14T17:42:37.726 INFO:tasks.cephadm:Stopping all daemons... 2024-02-14T17:42:37.726 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-14T17:42:37.726 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.a 2024-02-14T17:42:37.733 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:37 smithi027 podman[52959]: 2024-02-14 17:42:37.663268556 +0000 UTC m=+1.442310977 container remove c74a7ffc7aa9e13c044b7202524736aaf9fc6c7d8a3f84d187652cc52fcd0a6b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:38.046 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:37 smithi027 systemd[1]: Stopping Ceph mon.a for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:38.047 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:37 smithi027 podman[52959]: 2024-02-14 17:42:37.769103011 +0000 UTC m=+1.548145430 volume remove ea436f0a3d0bfa46b6b03dfebb66a6ae67e6208153530c78cd27f6644891b34d 2024-02-14T17:42:38.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:38 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a[25805]: 2024-02-14T17:42:38.045+0000 7fbf70e69700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:42:38.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:38 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a[25805]: 2024-02-14T17:42:38.045+0000 7fbf70e69700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-14T17:42:38.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:38 smithi027 podman[53133]: 2024-02-14 17:42:38.187777664 +0000 UTC m=+0.255117118 container died 6714e9fe1fa6c0df92ee908c354f6a2e47bd5850947446ec5a6ec2ad7bae8a0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-14T17:42:38.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:38 smithi027 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-14T17:42:38.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:38 smithi027 systemd[1]: Stopped Ceph alertmanager.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:42:38.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:38 smithi027 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@alertmanager.a.service: Consumed 1.055s CPU time. 2024-02-14T17:42:38.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:38 smithi027 systemd[1]: Starting Ceph alertmanager.a for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:38.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:38 smithi027 podman[53213]: 2024-02-14 17:42:38.186708603 +0000 UTC m=+0.018298504 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-14T17:42:38.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:38 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:42:38] ENGINE Bus STOPPING 2024-02-14T17:42:38.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:38 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:42:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-14T17:42:38.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:38 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:42:38] ENGINE Bus STOPPED 2024-02-14T17:42:38.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:38 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:42:38] ENGINE Bus STARTING 2024-02-14T17:42:38.485 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:38 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:42:38] ENGINE Serving on http://:::9283 2024-02-14T17:42:38.485 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:38 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: [14/Feb/2024:17:42:38] ENGINE Bus STARTED 2024-02-14T17:42:38.818 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:38 smithi027 podman[53133]: 2024-02-14 17:42:38.73129186 +0000 UTC m=+0.798631360 container cleanup 6714e9fe1fa6c0df92ee908c354f6a2e47bd5850947446ec5a6ec2ad7bae8a0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-14T17:42:38.819 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:38 smithi027 bash[53133]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a 2024-02-14T17:42:39.104 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:38 smithi027 podman[53213]: 2024-02-14 17:42:38.81794116 +0000 UTC m=+0.649531067 volume create 7e6174aafa06e545372dd77951e4f251aa4ac0efcd877245fd2c2c63699a682c 2024-02-14T17:42:39.104 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:38 smithi027 podman[53213]: 2024-02-14 17:42:38.901782975 +0000 UTC m=+0.733372875 container create b08fa6315521b231e3385a70e4107246448a86540ca1f90e5d114f4513ad2200 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:39.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:39 smithi027 podman[53213]: 2024-02-14 17:42:39.214639284 +0000 UTC m=+1.046229184 container init b08fa6315521b231e3385a70e4107246448a86540ca1f90e5d114f4513ad2200 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:39.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:39 smithi027 podman[53213]: 2024-02-14 17:42:39.220452348 +0000 UTC m=+1.052042251 container start b08fa6315521b231e3385a70e4107246448a86540ca1f90e5d114f4513ad2200 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:39.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:39 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[53258]: ts=2024-02-14T17:42:39.233Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-14T17:42:39.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:39 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[53258]: ts=2024-02-14T17:42:39.233Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-14T17:42:39.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:39 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[53258]: ts=2024-02-14T17:42:39.234Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.27 port=9094 2024-02-14T17:42:39.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:39 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[53258]: ts=2024-02-14T17:42:39.235Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-14T17:42:39.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:39 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[53258]: ts=2024-02-14T17:42:39.259Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-14T17:42:39.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:39 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[53258]: ts=2024-02-14T17:42:39.259Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-14T17:42:39.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:39 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[53258]: ts=2024-02-14T17:42:39.262Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-14T17:42:39.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:39 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[53258]: ts=2024-02-14T17:42:39.262Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-14T17:42:39.484 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:39 smithi027 bash[53213]: b08fa6315521b231e3385a70e4107246448a86540ca1f90e5d114f4513ad2200 2024-02-14T17:42:39.485 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:39 smithi027 systemd[1]: Started Ceph alertmanager.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:42:39.810 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:39 smithi027 podman[53225]: 2024-02-14 17:42:39.532969817 +0000 UTC m=+1.343465939 container remove 6714e9fe1fa6c0df92ee908c354f6a2e47bd5850947446ec5a6ec2ad7bae8a0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-a, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214) 2024-02-14T17:42:40.041 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.a.service' 2024-02-14T17:42:40.110 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:40 smithi027 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-14T17:42:40.110 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:40 smithi027 systemd[1]: Stopped Ceph mon.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:42:40.110 INFO:journalctl@ceph.mon.a.smithi027.stdout:Feb 14 17:42:40 smithi027 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.a.service: Consumed 11.349s CPU time. 2024-02-14T17:42:40.134 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:42:40.134 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-14T17:42:40.134 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-02-14T17:42:40.134 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.b 2024-02-14T17:42:40.479 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:40 smithi040 systemd[1]: Stopping Ceph mon.b for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:40.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:40 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-b[27548]: 2024-02-14T17:42:40.523+0000 7f45d4a63700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:42:40.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:40 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-b[27548]: 2024-02-14T17:42:40.523+0000 7f45d4a63700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-02-14T17:42:40.979 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:40 smithi040 podman[40709]: 2024-02-14 17:42:40.718900627 +0000 UTC m=+0.387309931 container died 544f97a89dd67c7cef5be4a9578afdf4cd9820d59c42fc93304d3f901d7ecc03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-02-14T17:42:41.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:41 smithi040 podman[40709]: 2024-02-14 17:42:41.096600237 +0000 UTC m=+0.765009533 container cleanup 544f97a89dd67c7cef5be4a9578afdf4cd9820d59c42fc93304d3f901d7ecc03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-14T17:42:41.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Feb 14 17:42:41 smithi040 bash[40709]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-b 2024-02-14T17:42:41.483 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:41 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[53258]: ts=2024-02-14T17:42:41.235Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00054592s 2024-02-14T17:42:41.519 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.b.service' 2024-02-14T17:42:41.605 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:42:41.605 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-02-14T17:42:41.606 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-02-14T17:42:41.606 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.c 2024-02-14T17:42:41.906 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:41 smithi096 systemd[1]: Stopping Ceph mon.c for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:42.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:41 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-c[27749]: 2024-02-14T17:42:41.903+0000 7f776e132700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:42:42.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:41 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-c[27749]: 2024-02-14T17:42:41.903+0000 7f776e132700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-02-14T17:42:42.295 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:42 smithi096 podman[40652]: 2024-02-14 17:42:42.08859818 +0000 UTC m=+0.289579124 container died ef7f0c7cea0a8ccaa91232dd383c00878012c2820be5e78e552b8eb3069fa65c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-14T17:42:42.695 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:42 smithi096 podman[40652]: 2024-02-14 17:42:42.434711393 +0000 UTC m=+0.635692333 container cleanup ef7f0c7cea0a8ccaa91232dd383c00878012c2820be5e78e552b8eb3069fa65c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-02-14T17:42:42.695 INFO:journalctl@ceph.mon.c.smithi096.stdout:Feb 14 17:42:42 smithi096 bash[40652]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mon-c 2024-02-14T17:42:42.846 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mon.c.service' 2024-02-14T17:42:42.890 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:42:42.890 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-02-14T17:42:42.890 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-14T17:42:42.890 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mgr.y 2024-02-14T17:42:43.229 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:42 smithi027 systemd[1]: Stopping Ceph mgr.y for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:43.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:43 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:42:43.227+0000 7ff0c863a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:42:43.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:43 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y[26088]: 2024-02-14T17:42:43.227+0000 7ff0c863a700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-14T17:42:43.483 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:43 smithi027 podman[53518]: 2024-02-14 17:42:43.258572531 +0000 UTC m=+0.165544437 container died 87bd103ac732c669a341530c99f54500b91e58fdf86c343e02822a0ce65e7cd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-14T17:42:43.960 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:43 smithi027 podman[53518]: 2024-02-14 17:42:43.632980966 +0000 UTC m=+0.539952872 container cleanup 87bd103ac732c669a341530c99f54500b91e58fdf86c343e02822a0ce65e7cd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-14T17:42:43.960 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Feb 14 17:42:43 smithi027 bash[53518]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-y 2024-02-14T17:42:44.105 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mgr.y.service' 2024-02-14T17:42:44.148 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:42:44.149 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-14T17:42:44.149 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-14T17:42:44.149 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mgr.x 2024-02-14T17:42:44.478 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:42:44 smithi040 systemd[1]: Stopping Ceph mgr.x for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:44.785 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:42:44 smithi040 podman[40838]: 2024-02-14 17:42:44.49043719 +0000 UTC m=+0.148693916 container died 9e3b48ef91d78206fec8838b2a85cf09d68e15d19133f77dafd9c34f64374c2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214) 2024-02-14T17:42:45.127 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:42:44 smithi040 podman[40838]: 2024-02-14 17:42:44.842147692 +0000 UTC m=+0.500404417 container cleanup 9e3b48ef91d78206fec8838b2a85cf09d68e15d19133f77dafd9c34f64374c2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-14T17:42:45.127 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Feb 14 17:42:44 smithi040 bash[40838]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-x 2024-02-14T17:42:45.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mgr.x.service' 2024-02-14T17:42:45.297 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:42:45.297 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-14T17:42:45.297 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-02-14T17:42:45.298 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mgr.z 2024-02-14T17:42:45.630 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:42:45 smithi096 systemd[1]: Stopping Ceph mgr.z for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:45.931 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:42:45 smithi096 podman[40785]: 2024-02-14 17:42:45.628599075 +0000 UTC m=+0.135509811 container died 9ec91534454f20920e063cbac18b64433de1b706a0bf80279255c3991edd6309 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:42:46.214 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:42:46 smithi096 podman[40785]: 2024-02-14 17:42:46.012842297 +0000 UTC m=+0.519753028 container cleanup 9ec91534454f20920e063cbac18b64433de1b706a0bf80279255c3991edd6309 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-14T17:42:46.214 INFO:journalctl@ceph.mgr.z.smithi096.stdout:Feb 14 17:42:46 smithi096 bash[40785]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-mgr-z 2024-02-14T17:42:46.327 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@mgr.z.service' 2024-02-14T17:42:46.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:42:46.363 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-02-14T17:42:46.363 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-14T17:42:46.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-932ae096-cb5f-11ee-95b9-87774f69a715@osd.0 2024-02-14T17:42:46.683 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:46 smithi027 systemd[1]: Stopping Ceph osd.0 for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:46.983 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:46 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0[38251]: 2024-02-14T17:42:46.681+0000 7f9927d18700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:42:46.983 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:46 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0[38251]: 2024-02-14T17:42:46.681+0000 7f9927d18700 -1 osd.0 31 *** Got signal Terminated *** 2024-02-14T17:42:46.983 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:46 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0[38251]: 2024-02-14T17:42:46.681+0000 7f9927d18700 -1 osd.0 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:42:49.483 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:42:49 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[53258]: ts=2024-02-14T17:42:49.237Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002097081s 2024-02-14T17:42:52.483 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:52 smithi027 podman[53766]: 2024-02-14 17:42:52.171689534 +0000 UTC m=+5.600217263 container died 5bf0e4876bafe515dcaf13c1b81e1458b64b130ffa5515d1b076863c981fe473 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-14T17:42:52.923 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:52 smithi027 podman[53766]: 2024-02-14 17:42:52.696135543 +0000 UTC m=+6.124663313 container cleanup 5bf0e4876bafe515dcaf13c1b81e1458b64b130ffa5515d1b076863c981fe473 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0) 2024-02-14T17:42:52.923 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:52 smithi027 bash[53766]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0 2024-02-14T17:42:53.233 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:52 smithi027 podman[54154]: 2024-02-14 17:42:52.922195076 +0000 UTC m=+0.744310128 container remove 5bf0e4876bafe515dcaf13c1b81e1458b64b130ffa5515d1b076863c981fe473 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-14T17:42:53.524 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:53 smithi027 podman[54251]: 2024-02-14 17:42:53.166620671 +0000 UTC m=+0.018582625 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:42:53.524 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:53 smithi027 podman[54251]: 2024-02-14 17:42:53.406547301 +0000 UTC m=+0.258509309 container create c390ff036a9d06f058baf4a55d5f78e754a9428d9591ba904640cc856f0f2513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-14T17:42:53.855 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:53 smithi027 podman[54251]: 2024-02-14 17:42:53.642992902 +0000 UTC m=+0.494954851 container init c390ff036a9d06f058baf4a55d5f78e754a9428d9591ba904640cc856f0f2513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-14T17:42:53.855 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:53 smithi027 podman[54251]: 2024-02-14 17:42:53.648834921 +0000 UTC m=+0.500796870 container start c390ff036a9d06f058baf4a55d5f78e754a9428d9591ba904640cc856f0f2513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-14T17:42:54.154 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:53 smithi027 podman[54251]: 2024-02-14 17:42:53.93615099 +0000 UTC m=+0.788112937 container attach c390ff036a9d06f058baf4a55d5f78e754a9428d9591ba904640cc856f0f2513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux ) 2024-02-14T17:42:54.483 INFO:journalctl@ceph.osd.0.smithi027.stdout:Feb 14 17:42:54 smithi027 podman[54251]: 2024-02-14 17:42:54.394885158 +0000 UTC m=+1.246847098 container died c390ff036a9d06f058baf4a55d5f78e754a9428d9591ba904640cc856f0f2513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240214, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-14T17:42:54.982 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@osd.0.service' 2024-02-14T17:42:55.038 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:42:55.038 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-14T17:42:55.038 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-14T17:42:55.039 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-932ae096-cb5f-11ee-95b9-87774f69a715@osd.1 2024-02-14T17:42:55.383 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:42:55 smithi040 systemd[1]: Stopping Ceph osd.1 for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:42:56.186 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:42:55 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1[32850]: 2024-02-14T17:42:55.904+0000 7feb9c006700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:42:56.187 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:42:55 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1[32850]: 2024-02-14T17:42:55.904+0000 7feb9c006700 -1 osd.1 31 *** Got signal Terminated *** 2024-02-14T17:42:56.187 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:42:55 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1[32850]: 2024-02-14T17:42:55.904+0000 7feb9c006700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:43:01.645 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:43:01 smithi040 podman[41401]: 2024-02-14 17:43:01.309798513 +0000 UTC m=+5.834529013 container died 9b75dab64136faf96d906c59f31bf50bb707b3f26fbff202bf31f98f19cc4961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-14T17:43:01.646 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:43:01 smithi040 podman[41401]: 2024-02-14 17:43:01.644884164 +0000 UTC m=+6.169614664 container cleanup 9b75dab64136faf96d906c59f31bf50bb707b3f26fbff202bf31f98f19cc4961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1, org.label-schema.build-date=20240214, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:43:01.978 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:43:01 smithi040 bash[41401]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1 2024-02-14T17:43:01.979 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:43:01 smithi040 podman[41719]: 2024-02-14 17:43:01.812844857 +0000 UTC m=+0.505888753 container remove 9b75dab64136faf96d906c59f31bf50bb707b3f26fbff202bf31f98f19cc4961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:43:02.479 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:43:02 smithi040 podman[41816]: 2024-02-14 17:43:02.213555844 +0000 UTC m=+0.112004520 container create 906b3b72a8f76d02dd9ed898a68f1cd2af30096daf90a2ea700e9ce11d189ecc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1-deactivate, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-02-14T17:43:02.479 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:43:02 smithi040 podman[41816]: 2024-02-14 17:43:02.11935008 +0000 UTC m=+0.017798764 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:43:02.479 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:43:02 smithi040 podman[41816]: 2024-02-14 17:43:02.41567357 +0000 UTC m=+0.314122251 container init 906b3b72a8f76d02dd9ed898a68f1cd2af30096daf90a2ea700e9ce11d189ecc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_CLEAN=True) 2024-02-14T17:43:02.479 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:43:02 smithi040 podman[41816]: 2024-02-14 17:43:02.42085556 +0000 UTC m=+0.319304241 container start 906b3b72a8f76d02dd9ed898a68f1cd2af30096daf90a2ea700e9ce11d189ecc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-14T17:43:02.979 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:43:02 smithi040 podman[41816]: 2024-02-14 17:43:02.498814174 +0000 UTC m=+0.397262864 container attach 906b3b72a8f76d02dd9ed898a68f1cd2af30096daf90a2ea700e9ce11d189ecc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-14T17:43:02.979 INFO:journalctl@ceph.osd.1.smithi040.stdout:Feb 14 17:43:02 smithi040 podman[41816]: 2024-02-14 17:43:02.71820664 +0000 UTC m=+0.616655322 container died 906b3b72a8f76d02dd9ed898a68f1cd2af30096daf90a2ea700e9ce11d189ecc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:43:03.350 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@osd.1.service' 2024-02-14T17:43:03.393 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:43:03.393 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-14T17:43:03.393 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-14T17:43:03.393 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-932ae096-cb5f-11ee-95b9-87774f69a715@osd.2 2024-02-14T17:43:03.694 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:03 smithi096 systemd[1]: Stopping Ceph osd.2 for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:43:04.045 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:03 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2[34218]: 2024-02-14T17:43:03.692+0000 7f2121f91700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:43:04.045 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:03 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2[34218]: 2024-02-14T17:43:03.692+0000 7f2121f91700 -1 osd.2 31 *** Got signal Terminated *** 2024-02-14T17:43:04.045 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:03 smithi096 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2[34218]: 2024-02-14T17:43:03.692+0000 7f2121f91700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:43:09.423 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:09 smithi096 podman[41487]: 2024-02-14 17:43:09.131054714 +0000 UTC m=+5.536954047 container died c53d1a6d5381eb3dc84781015236df2929b32b2350c9508f6ff1a03ea95a6b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-02-14T17:43:09.424 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:09 smithi096 podman[41487]: 2024-02-14 17:43:09.422894388 +0000 UTC m=+5.828793728 container cleanup c53d1a6d5381eb3dc84781015236df2929b32b2350c9508f6ff1a03ea95a6b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-14T17:43:09.795 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:09 smithi096 bash[41487]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2 2024-02-14T17:43:09.795 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:09 smithi096 podman[41509]: 2024-02-14 17:43:09.657103157 +0000 UTC m=+0.523743050 container remove c53d1a6d5381eb3dc84781015236df2929b32b2350c9508f6ff1a03ea95a6b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-14T17:43:10.295 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:09 smithi096 podman[41610]: 2024-02-14 17:43:09.893056754 +0000 UTC m=+0.017938480 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:43:10.296 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:10 smithi096 podman[41610]: 2024-02-14 17:43:10.201361386 +0000 UTC m=+0.326243099 container create 6601eeda459b4e63893f59aba71f48329462e4ee68f8623c35d10312b8cb9243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-02-14T17:43:10.613 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:10 smithi096 podman[41610]: 2024-02-14 17:43:10.529126366 +0000 UTC m=+0.654008081 container init 6601eeda459b4e63893f59aba71f48329462e4ee68f8623c35d10312b8cb9243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240214) 2024-02-14T17:43:10.613 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:10 smithi096 podman[41610]: 2024-02-14 17:43:10.534842111 +0000 UTC m=+0.659723824 container start 6601eeda459b4e63893f59aba71f48329462e4ee68f8623c35d10312b8cb9243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=) 2024-02-14T17:43:11.200 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:10 smithi096 podman[41610]: 2024-02-14 17:43:10.94709847 +0000 UTC m=+1.071980197 container attach 6601eeda459b4e63893f59aba71f48329462e4ee68f8623c35d10312b8cb9243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-14T17:43:11.200 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:10 smithi096 podman[41698]: 2024-02-14 17:43:10.94785598 +0000 UTC m=+0.108970474 container died 6601eeda459b4e63893f59aba71f48329462e4ee68f8623c35d10312b8cb9243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-14T17:43:11.453 INFO:journalctl@ceph.osd.2.smithi096.stdout:Feb 14 17:43:11 smithi096 podman[41698]: 2024-02-14 17:43:11.441159404 +0000 UTC m=+0.602273905 container remove 6601eeda459b4e63893f59aba71f48329462e4ee68f8623c35d10312b8cb9243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-14T17:43:11.458 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@osd.2.service' 2024-02-14T17:43:11.499 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:43:11.499 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-14T17:43:11.499 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-14T17:43:11.499 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-932ae096-cb5f-11ee-95b9-87774f69a715@rgw.foo.a 2024-02-14T17:43:11.831 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:43:11 smithi027 systemd[1]: Stopping Ceph rgw.foo.a for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:43:12.112 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:43:11 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-rgw-foo-a[41161]: 2024-02-14T17:43:11.828+0000 7f39d658c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:43:12.112 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:43:11 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-rgw-foo-a[41161]: 2024-02-14T17:43:11.828+0000 7f39e4dfdac0 -1 shutting down 2024-02-14T17:43:22.165 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:43:21 smithi027 bash[55211]: time="2024-02-14T17:43:21Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-932ae096-cb5f-11ee-95b9-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-14T17:43:22.165 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:43:21 smithi027 podman[55211]: 2024-02-14 17:43:21.853782155 +0000 UTC m=+10.151984892 container died 77b6b21cb6a8e1052f332558a9d47a0618085ee47004e70c2e526ccaa0e136d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-rgw-foo-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-14T17:43:22.483 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:43:22 smithi027 podman[55211]: 2024-02-14 17:43:22.337951649 +0000 UTC m=+10.636154429 container cleanup 77b6b21cb6a8e1052f332558a9d47a0618085ee47004e70c2e526ccaa0e136d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214) 2024-02-14T17:43:22.483 INFO:journalctl@ceph.rgw.foo.a.smithi027.stdout:Feb 14 17:43:22 smithi027 bash[55211]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-rgw-foo-a 2024-02-14T17:43:22.714 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@rgw.foo.a.service' 2024-02-14T17:43:22.729 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:22.350Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.27:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.27:8765: connect: connection refused" 2024-02-14T17:43:22.729 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:22 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:22.350Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.27:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.27:8765: connect: connection refused" 2024-02-14T17:43:22.756 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:43:22.756 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-14T17:43:22.757 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-14T17:43:22.757 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-932ae096-cb5f-11ee-95b9-87774f69a715@prometheus.a 2024-02-14T17:43:23.080 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:22 smithi040 systemd[1]: Stopping Ceph prometheus.a for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:43:23.080 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:23.078Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:43:23.080 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:23.079Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-14T17:43:23.369 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:23.079Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-14T17:43:23.370 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:23.079Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-14T17:43:23.370 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:23.079Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-14T17:43:23.370 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:23.079Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-14T17:43:23.370 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:23.079Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-14T17:43:23.370 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:23.079Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-14T17:43:23.370 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:23.079Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-14T17:43:23.370 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:23.079Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-14T17:43:23.370 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:23.079Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-14T17:43:23.370 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a[35552]: ts=2024-02-14T17:43:23.079Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-14T17:43:23.370 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 podman[42616]: 2024-02-14 17:43:23.089434272 +0000 UTC m=+0.126034575 container died 1f71374f1913cfb359b000f830d6fa43662db0b7577d95d221633f8e1afa3e4f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:43:23.686 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 podman[42616]: 2024-02-14 17:43:23.467698911 +0000 UTC m=+0.504299212 container cleanup 1f71374f1913cfb359b000f830d6fa43662db0b7577d95d221633f8e1afa3e4f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:43:23.686 INFO:journalctl@ceph.prometheus.a.smithi040.stdout:Feb 14 17:43:23 smithi040 bash[42616]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-prometheus-a 2024-02-14T17:43:23.802 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-932ae096-cb5f-11ee-95b9-87774f69a715@prometheus.a.service' 2024-02-14T17:43:23.839 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:43:23.840 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-14T17:43:23.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-14T17:43:24.042 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:43:25.233 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:43:25 smithi027 systemd[1]: Stopping Ceph alertmanager.a for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:43:25.678 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:43:25 smithi027 ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a[53258]: ts=2024-02-14T17:43:25.370Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:43:25.678 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:43:25 smithi027 podman[55616]: 2024-02-14 17:43:25.390911803 +0000 UTC m=+0.133358578 container died b08fa6315521b231e3385a70e4107246448a86540ca1f90e5d114f4513ad2200 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:43:25.983 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:43:25 smithi027 podman[55616]: 2024-02-14 17:43:25.858979118 +0000 UTC m=+0.601425892 container cleanup b08fa6315521b231e3385a70e4107246448a86540ca1f90e5d114f4513ad2200 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:43:25.983 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:43:25 smithi027 bash[55616]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a 2024-02-14T17:43:26.416 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:43:26 smithi027 podman[55635]: 2024-02-14 17:43:26.145412348 +0000 UTC m=+0.756893809 container remove b08fa6315521b231e3385a70e4107246448a86540ca1f90e5d114f4513ad2200 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:43:26.417 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:43:26 smithi027 podman[55635]: 2024-02-14 17:43:26.227739371 +0000 UTC m=+0.839220825 volume remove 7e6174aafa06e545372dd77951e4f251aa4ac0efcd877245fd2c2c63699a682c 2024-02-14T17:43:26.733 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:43:26 smithi027 systemd[1]: Stopping Ceph node-exporter.a for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:43:26.733 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:43:26 smithi027 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-14T17:43:26.733 INFO:journalctl@ceph.alertmanager.a.smithi027.stdout:Feb 14 17:43:26 smithi027 systemd[1]: Stopped Ceph alertmanager.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:43:27.233 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:43:26 smithi027 podman[55752]: 2024-02-14 17:43:26.877241615 +0000 UTC m=+0.106659704 container died fffd927061487782ca1811a5bdacc2bb36677d9a82f7b29375e1ecbfbb2cd247 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:43:27.719 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:43:27 smithi027 podman[55752]: 2024-02-14 17:43:27.27187113 +0000 UTC m=+0.501289224 container cleanup fffd927061487782ca1811a5bdacc2bb36677d9a82f7b29375e1ecbfbb2cd247 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:43:27.719 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:43:27 smithi027 bash[55752]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a 2024-02-14T17:43:27.719 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:43:27 smithi027 podman[55771]: 2024-02-14 17:43:27.473115572 +0000 UTC m=+0.600674668 container remove fffd927061487782ca1811a5bdacc2bb36677d9a82f7b29375e1ecbfbb2cd247 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:43:27.719 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:43:27 smithi027 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:43:27.983 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:43:27 smithi027 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-14T17:43:27.984 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:43:27 smithi027 systemd[1]: Stopped Ceph node-exporter.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:43:27.984 INFO:journalctl@ceph.node-exporter.a.smithi027.stdout:Feb 14 17:43:27 smithi027 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@node-exporter.a.service: Consumed 1.954s CPU time. 2024-02-14T17:43:28.411 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-14T17:43:30.200 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:43:31.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:43:31 smithi040 systemd[1]: Stopping Ceph grafana.a for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:43:31.760 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:43:31 smithi040 podman[43227]: 2024-02-14 17:43:31.505708851 +0000 UTC m=+0.118875311 container died c8a0478db7db24db66774648fbf1b6cb271854285ec8c691e39e80d64cfc264c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, version=8.5, architecture=x86_64, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc.) 2024-02-14T17:43:32.111 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:43:31 smithi040 podman[43227]: 2024-02-14 17:43:31.850228942 +0000 UTC m=+0.463395401 container cleanup c8a0478db7db24db66774648fbf1b6cb271854285ec8c691e39e80d64cfc264c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., version=8.5, description=Ceph Grafana Container, maintainer=Paul Cuzner , name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, io.openshift.expose-services=, vcs-type=git, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-14T17:43:32.111 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:43:31 smithi040 bash[43227]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a 2024-02-14T17:43:32.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:43:32 smithi040 podman[43247]: 2024-02-14 17:43:32.109710161 +0000 UTC m=+0.606917385 container remove c8a0478db7db24db66774648fbf1b6cb271854285ec8c691e39e80d64cfc264c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, distribution-scope=public, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, vcs-type=git, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, description=Ceph Grafana Container, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., maintainer=Paul Cuzner , name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-02-14T17:43:32.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:43:32 smithi040 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:43:32.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:43:32 smithi040 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-14T17:43:32.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:43:32 smithi040 systemd[1]: Stopped Ceph grafana.a for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:43:32.479 INFO:journalctl@ceph.grafana.a.smithi040.stdout:Feb 14 17:43:32 smithi040 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@grafana.a.service: Consumed 3.836s CPU time. 2024-02-14T17:43:32.743 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:43:32 smithi040 systemd[1]: Stopping Ceph node-exporter.b for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:43:33.064 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:43:32 smithi040 podman[43365]: 2024-02-14 17:43:32.74285496 +0000 UTC m=+0.127726213 container died 9ef0ae1a5738346c9d1a3451413be3279f76b24b6fbcb317af1fe5cbb7e92007 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:43:33.464 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:43:33 smithi040 podman[43365]: 2024-02-14 17:43:33.196186535 +0000 UTC m=+0.581057779 container cleanup 9ef0ae1a5738346c9d1a3451413be3279f76b24b6fbcb317af1fe5cbb7e92007 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:43:33.464 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:43:33 smithi040 bash[43365]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b 2024-02-14T17:43:33.729 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:43:33 smithi040 podman[43384]: 2024-02-14 17:43:33.463627682 +0000 UTC m=+0.721937918 container remove 9ef0ae1a5738346c9d1a3451413be3279f76b24b6fbcb317af1fe5cbb7e92007 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:43:33.729 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:43:33 smithi040 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:43:33.729 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:43:33 smithi040 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-14T17:43:33.729 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:43:33 smithi040 systemd[1]: Stopped Ceph node-exporter.b for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:43:33.729 INFO:journalctl@ceph.node-exporter.b.smithi040.stdout:Feb 14 17:43:33 smithi040 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@node-exporter.b.service: Consumed 1.803s CPU time. 2024-02-14T17:43:34.248 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-14T17:43:34.430 INFO:teuthology.orchestra.run.smithi096.stdout:Deleting cluster with fsid: 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:43:35.518 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:43:35 smithi096 systemd[1]: Stopping Ceph node-exporter.c for 932ae096-cb5f-11ee-95b9-87774f69a715... 2024-02-14T17:43:35.788 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:43:35 smithi096 podman[42689]: 2024-02-14 17:43:35.517181663 +0000 UTC m=+0.116163229 container died 05081f4ee8f0ed8570ce7346f1c917917a4bc17573c63bf999392bf7650d39dd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-02-14T17:43:36.045 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:43:35 smithi096 podman[42689]: 2024-02-14 17:43:35.870168236 +0000 UTC m=+0.469149795 container cleanup 05081f4ee8f0ed8570ce7346f1c917917a4bc17573c63bf999392bf7650d39dd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-02-14T17:43:36.045 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:43:35 smithi096 bash[42689]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c 2024-02-14T17:43:36.435 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:43:36 smithi096 podman[42707]: 2024-02-14 17:43:36.137633124 +0000 UTC m=+0.625414392 container remove 05081f4ee8f0ed8570ce7346f1c917917a4bc17573c63bf999392bf7650d39dd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-932ae096-cb5f-11ee-95b9-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-02-14T17:43:36.435 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:43:36 smithi096 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@node-exporter.c.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:43:36.435 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:43:36 smithi096 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@node-exporter.c.service: Failed with result 'exit-code'. 2024-02-14T17:43:36.435 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:43:36 smithi096 systemd[1]: Stopped Ceph node-exporter.c for 932ae096-cb5f-11ee-95b9-87774f69a715. 2024-02-14T17:43:36.435 INFO:journalctl@ceph.node-exporter.c.smithi096.stdout:Feb 14 17:43:36 smithi096 systemd[1]: ceph-932ae096-cb5f-11ee-95b9-87774f69a715@node-exporter.c.service: Consumed 1.784s CPU time. 2024-02-14T17:43:36.914 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:43:36.954 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:43:36.989 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:43:37.018 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-14T17:43:37.020 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809/remote/smithi027/crash 2024-02-14T17:43:37.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/crash -- . 2024-02-14T17:43:37.071 INFO:teuthology.orchestra.run.smithi027.stderr:tar: /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-14T17:43:37.071 INFO:teuthology.orchestra.run.smithi027.stderr:tar: Error is not recoverable: exiting now 2024-02-14T17:43:37.073 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809/remote/smithi040/crash 2024-02-14T17:43:37.073 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/crash -- . 2024-02-14T17:43:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:tar: /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-14T17:43:37.118 INFO:teuthology.orchestra.run.smithi040.stderr:tar: Error is not recoverable: exiting now 2024-02-14T17:43:37.120 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809/remote/smithi096/crash 2024-02-14T17:43:37.120 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/crash -- . 2024-02-14T17:43:37.159 INFO:teuthology.orchestra.run.smithi096.stderr:tar: /var/lib/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-14T17:43:37.159 INFO:teuthology.orchestra.run.smithi096.stderr:tar: Error is not recoverable: exiting now 2024-02-14T17:43:37.160 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-14T17:43:37.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-14T17:43:37.196 INFO:teuthology.orchestra.run.smithi027.stdout:2024-02-14T17:39:16.096978+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-14T17:43:37.197 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-14T17:43:37.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[SEC\]' /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-14T17:43:37.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]' /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-14T17:43:37.347 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[WRN\]' /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-14T17:43:37.421 INFO:teuthology.orchestra.run.smithi027.stdout:2024-02-14T17:39:16.096978+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-14T17:43:37.421 INFO:tasks.cephadm:Compressing logs... 2024-02-14T17:43:37.421 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-14T17:43:37.463 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-14T17:43:37.466 DEBUG:teuthology.orchestra.run.smithi096:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-14T17:43:37.492 INFO:teuthology.orchestra.run.smithi096.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-14T17:43:37.493 INFO:teuthology.orchestra.run.smithi096.stderr:: No such file or directory 2024-02-14T17:43:37.493 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-14T17:43:37.493 INFO:teuthology.orchestra.run.smithi096.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mgr.z.log 2024-02-14T17:43:37.493 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-14T17:43:37.496 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.audit.log: /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.log 2024-02-14T17:43:37.496 INFO:teuthology.orchestra.run.smithi096.stderr:gzip/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.cephadm.log: -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-osd.2.log 2024-02-14T17:43:37.496 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.log: 81.8% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-14T17:43:37.497 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-14T17:43:37.497 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mon.c.log 2024-02-14T17:43:37.497 INFO:teuthology.orchestra.run.smithi096.stderr: 90.1% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-14T17:43:37.497 INFO:teuthology.orchestra.run.smithi096.stderr: 84.8% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-14T17:43:37.497 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-volume.log: 79.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-14T17:43:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-14T17:43:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:: No such file or directory 2024-02-14T17:43:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mon.a.log 2024-02-14T17:43:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-14T17:43:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-client.rgw.foo.a.log 2024-02-14T17:43:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mgr.y.log 2024-02-14T17:43:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-osd.0.log 2024-02-14T17:43:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mgr.y.log: 57.9% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-02-14T17:43:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-14T17:43:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.log 2024-02-14T17:43:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-14T17:43:37.499 INFO:teuthology.orchestra.run.smithi027.stderr: 89.9% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.audit.log.gz/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.log: 2024-02-14T17:43:37.499 INFO:teuthology.orchestra.run.smithi027.stderr: 83.0% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-14T17:43:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-volume.log: 85.2% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-14T17:43:37.499 INFO:teuthology.orchestra.run.smithi027.stderr: 84.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-14T17:43:37.500 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mon.c.log: 90.2% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mgr.z.log.gz 2024-02-14T17:43:37.502 INFO:teuthology.orchestra.run.smithi040.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-14T17:43:37.502 INFO:teuthology.orchestra.run.smithi040.stderr:: No such file or directory 2024-02-14T17:43:37.502 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-14T17:43:37.503 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-osd.1.log 2024-02-14T17:43:37.503 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mgr.x.log 2024-02-14T17:43:37.503 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-14T17:43:37.503 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.log 2024-02-14T17:43:37.503 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mgr.x.log: /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-14T17:43:37.503 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.log: 81.8% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-14T17:43:37.503 INFO:teuthology.orchestra.run.smithi040.stderr:gzip 90.0% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-14T17:43:37.503 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mon.b.log 2024-02-14T17:43:37.503 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-volume.log: 84.6% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-14T17:43:37.503 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mon.b.log: 77.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-14T17:43:37.503 INFO:teuthology.orchestra.run.smithi040.stderr: 90.3% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mgr.x.log.gz 2024-02-14T17:43:37.511 INFO:teuthology.orchestra.run.smithi096.stderr: 90.2% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-14T17:43:37.512 INFO:teuthology.orchestra.run.smithi027.stderr: 90.1% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-14T17:43:37.515 INFO:teuthology.orchestra.run.smithi040.stderr: 90.1% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-14T17:43:37.564 INFO:teuthology.orchestra.run.smithi040.stderr: 92.4% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mon.b.log.gz 2024-02-14T17:43:37.566 INFO:teuthology.orchestra.run.smithi096.stderr: 92.3% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mon.c.log.gz 2024-02-14T17:43:37.581 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mgr.y.log.gz 2024-02-14T17:43:37.731 INFO:teuthology.orchestra.run.smithi027.stderr: 91.6% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-mon.a.log.gz 2024-02-14T17:43:37.767 INFO:teuthology.orchestra.run.smithi096.stderr: 94.3% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-osd.2.log.gz 2024-02-14T17:43:37.769 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-02-14T17:43:37.769 INFO:teuthology.orchestra.run.smithi096.stderr:real 0m0.289s 2024-02-14T17:43:37.769 INFO:teuthology.orchestra.run.smithi096.stderr:user 0m0.364s 2024-02-14T17:43:37.769 INFO:teuthology.orchestra.run.smithi096.stderr:sys 0m0.039s 2024-02-14T17:43:37.802 INFO:teuthology.orchestra.run.smithi027.stderr: 94.3% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-osd.0.log.gz 2024-02-14T17:43:37.804 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-02-14T17:43:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:real 0m0.328s 2024-02-14T17:43:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:user 0m0.643s 2024-02-14T17:43:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m0.052s 2024-02-14T17:43:37.830 INFO:teuthology.orchestra.run.smithi040.stderr: 94.3% -- replaced with /var/log/ceph/932ae096-cb5f-11ee-95b9-87774f69a715/ceph-osd.1.log.gz 2024-02-14T17:43:37.832 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-02-14T17:43:37.832 INFO:teuthology.orchestra.run.smithi040.stderr:real 0m0.352s 2024-02-14T17:43:37.832 INFO:teuthology.orchestra.run.smithi040.stderr:user 0m0.426s 2024-02-14T17:43:37.832 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m0.041s 2024-02-14T17:43:37.833 INFO:tasks.cephadm:Archiving logs... 2024-02-14T17:43:37.833 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809/remote/smithi027/log 2024-02-14T17:43:37.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-14T17:43:37.990 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809/remote/smithi040/log 2024-02-14T17:43:37.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-14T17:43:38.094 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809/remote/smithi096/log 2024-02-14T17:43:38.095 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-14T17:43:38.191 INFO:tasks.cephadm:Removing cluster... 2024-02-14T17:43:38.191 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 --force 2024-02-14T17:43:38.395 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:43:38.513 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 --force 2024-02-14T17:43:38.701 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:43:38.813 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 932ae096-cb5f-11ee-95b9-87774f69a715 --force 2024-02-14T17:43:38.999 INFO:teuthology.orchestra.run.smithi096.stdout:Deleting cluster with fsid: 932ae096-cb5f-11ee-95b9-87774f69a715 2024-02-14T17:43:39.113 INFO:tasks.cephadm:Removing cephadm ... 2024-02-14T17:43:39.113 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-14T17:43:39.130 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-14T17:43:39.146 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-14T17:43:39.161 INFO:tasks.cephadm:Teardown complete 2024-02-14T17:43:39.161 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-02-14T17:43:39.173 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi027:/dev/vg_nvme/lv_1... 2024-02-14T17:43:39.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo nvme disconnect -n lv_1 2024-02-14T17:43:39.319 INFO:teuthology.orchestra.run.smithi027.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-02-14T17:43:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:39.321 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi027:/dev/vg_nvme/lv_2... 2024-02-14T17:43:39.321 DEBUG:teuthology.orchestra.run.smithi027:> sudo nvme disconnect -n lv_2 2024-02-14T17:43:39.481 INFO:teuthology.orchestra.run.smithi027.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-02-14T17:43:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:39.483 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi027:/dev/vg_nvme/lv_3... 2024-02-14T17:43:39.483 DEBUG:teuthology.orchestra.run.smithi027:> sudo nvme disconnect -n lv_3 2024-02-14T17:43:39.668 INFO:teuthology.orchestra.run.smithi027.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-02-14T17:43:39.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:39.669 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi027:/dev/vg_nvme/lv_4... 2024-02-14T17:43:39.670 DEBUG:teuthology.orchestra.run.smithi027:> sudo nvme disconnect -n lv_4 2024-02-14T17:43:39.834 INFO:teuthology.orchestra.run.smithi027.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-02-14T17:43:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:39.835 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-02-14T17:43:39.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/scratch_devs 2024-02-14T17:43:39.867 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi040:/dev/vg_nvme/lv_1... 2024-02-14T17:43:39.867 DEBUG:teuthology.orchestra.run.smithi040:> sudo nvme disconnect -n lv_1 2024-02-14T17:43:40.027 INFO:teuthology.orchestra.run.smithi040.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-02-14T17:43:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:40.029 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi040:/dev/vg_nvme/lv_2... 2024-02-14T17:43:40.029 DEBUG:teuthology.orchestra.run.smithi040:> sudo nvme disconnect -n lv_2 2024-02-14T17:43:40.159 INFO:teuthology.orchestra.run.smithi040.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-02-14T17:43:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:40.161 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi040:/dev/vg_nvme/lv_3... 2024-02-14T17:43:40.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo nvme disconnect -n lv_3 2024-02-14T17:43:40.308 INFO:teuthology.orchestra.run.smithi040.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-02-14T17:43:40.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:40.310 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi040:/dev/vg_nvme/lv_4... 2024-02-14T17:43:40.310 DEBUG:teuthology.orchestra.run.smithi040:> sudo nvme disconnect -n lv_4 2024-02-14T17:43:40.436 INFO:teuthology.orchestra.run.smithi040.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-02-14T17:43:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:40.438 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-02-14T17:43:40.438 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/scratch_devs 2024-02-14T17:43:40.466 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi096:/dev/vg_nvme/lv_1... 2024-02-14T17:43:40.466 DEBUG:teuthology.orchestra.run.smithi096:> sudo nvme disconnect -n lv_1 2024-02-14T17:43:40.618 INFO:teuthology.orchestra.run.smithi096.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-02-14T17:43:40.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:40.620 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi096:/dev/vg_nvme/lv_2... 2024-02-14T17:43:40.620 DEBUG:teuthology.orchestra.run.smithi096:> sudo nvme disconnect -n lv_2 2024-02-14T17:43:40.767 INFO:teuthology.orchestra.run.smithi096.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-02-14T17:43:40.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:40.769 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi096:/dev/vg_nvme/lv_3... 2024-02-14T17:43:40.769 DEBUG:teuthology.orchestra.run.smithi096:> sudo nvme disconnect -n lv_3 2024-02-14T17:43:40.920 INFO:teuthology.orchestra.run.smithi096.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-02-14T17:43:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:40.921 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi096:/dev/vg_nvme/lv_4... 2024-02-14T17:43:40.921 DEBUG:teuthology.orchestra.run.smithi096:> sudo nvme disconnect -n lv_4 2024-02-14T17:43:41.074 INFO:teuthology.orchestra.run.smithi096.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-02-14T17:43:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:41.076 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-02-14T17:43:41.076 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/scratch_devs 2024-02-14T17:43:41.106 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-14T17:43:41.115 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-14T17:43:41.116 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-14T17:43:41.118 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-14T17:43:41.121 DEBUG:teuthology.orchestra.run.smithi096:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-14T17:43:41.131 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-02-14T17:43:41.135 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-02-14T17:43:41.135 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-14T17:43:41.136 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-02-14T17:43:41.136 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +2482us[+2482us] +/- 148ms 2024-02-14T17:43:41.136 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -551us[ -552us] +/- 74ms 2024-02-14T17:43:41.136 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -557us[ -557us] +/- 62ms 2024-02-14T17:43:41.136 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:43:41.139 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-14T17:43:41.139 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-02-14T17:43:41.139 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 +2489us[+2490us] +/- 148ms 2024-02-14T17:43:41.139 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -529us[ -528us] +/- 74ms 2024-02-14T17:43:41.139 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 -557us[ -556us] +/- 62ms 2024-02-14T17:43:41.140 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:43:41.162 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-02-14T17:43:41.166 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-14T17:43:41.166 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-02-14T17:43:41.166 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +2479us[+2479us] +/- 148ms 2024-02-14T17:43:41.166 INFO:teuthology.orchestra.run.smithi096.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -551us[ -559us] +/- 74ms 2024-02-14T17:43:41.166 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -548us[ -556us] +/- 62ms 2024-02-14T17:43:41.167 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:43:41.167 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-14T17:43:41.179 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-14T17:43:41.180 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-14T17:43:41.196 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-14T17:43:41.280 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-14T17:43:41.356 DEBUG:teuthology.orchestra.run.smithi096:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-14T17:43:41.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-14T17:43:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:41.467 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-14T17:43:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:41.499 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-14T17:43:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:41.531 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-14T17:43:41.541 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-14T17:43:41.594 INFO:teuthology.task.internal:Duration was 714.001589 seconds 2024-02-14T17:43:41.594 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-14T17:43:41.631 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-14T17:43:41.632 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-14T17:43:41.634 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-14T17:43:41.637 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-14T17:43:41.681 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-14T17:43:41.682 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-14T17:43:41.683 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-14T17:43:42.084 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-14T17:43:42.084 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-02-14T17:43:42.084 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-14T17:43:42.105 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-02-14T17:43:42.106 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-14T17:43:42.129 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi096.front.sepia.ceph.com 2024-02-14T17:43:42.130 DEBUG:teuthology.orchestra.run.smithi096:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-14T17:43:42.153 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-14T17:43:42.154 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-14T17:43:42.156 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-14T17:43:42.172 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-14T17:43:42.224 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-14T17:43:42.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-14T17:43:42.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-14T17:43:42.244 DEBUG:teuthology.orchestra.run.smithi096:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-14T17:43:42.427 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-14T17:43:42.443 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-14T17:43:42.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-14T17:43:42.475 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-14T17:43:42.505 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-14T17:43:42.532 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-14T17:43:42.546 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-14T17:43:42.548 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-14T17:43:42.551 DEBUG:teuthology.orchestra.run.smithi096:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-14T17:43:42.574 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-02-14T17:43:42.575 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-02-14T17:43:42.598 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = core 2024-02-14T17:43:42.634 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-14T17:43:42.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:42.656 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-14T17:43:42.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:42.671 DEBUG:teuthology.orchestra.run.smithi096:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-14T17:43:42.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:43:42.689 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-14T17:43:42.702 INFO:teuthology.task.internal:Transferring archived files... 2024-02-14T17:43:42.703 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809/remote/smithi027 2024-02-14T17:43:42.703 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-14T17:43:42.746 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809/remote/smithi040 2024-02-14T17:43:42.746 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-14T17:43:42.782 DEBUG:teuthology.misc:Transferring archived files from smithi096:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809/remote/smithi096 2024-02-14T17:43:42.782 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-14T17:43:42.819 INFO:teuthology.task.internal:Removing archive directory... 2024-02-14T17:43:42.819 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-14T17:43:42.822 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-14T17:43:42.824 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-14T17:43:42.868 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-14T17:43:42.880 INFO:teuthology.task.internal:Not uploading archives. 2024-02-14T17:43:42.880 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-14T17:43:42.892 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-14T17:43:42.892 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-14T17:43:42.894 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-14T17:43:42.896 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-14T17:43:42.906 INFO:teuthology.orchestra.run.smithi027.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 14 17:43 /home/ubuntu/cephtest 2024-02-14T17:43:42.908 INFO:teuthology.orchestra.run.smithi040.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 14 17:43 /home/ubuntu/cephtest 2024-02-14T17:43:42.925 INFO:teuthology.orchestra.run.smithi096.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 14 17:43 /home/ubuntu/cephtest 2024-02-14T17:43:42.927 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-14T17:43:42.938 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-14T17:43:42.961 INFO:teuthology.nuke:Checking targets against current locks 2024-02-14T17:43:42.989 DEBUG:teuthology.nuke:shortname: smithi027 2024-02-14T17:43:42.989 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-14T17:43:43.014 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-14 17:21:36.119003', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBRO4n+A/9PeQeviwJ8onInkR4bocXJa/AukutL0IlqcwSH5AO8WvTkxDW5eSyfThakbTjMVv2yuIAa6kdxoItI='} 2024-02-14T17:43:43.018 DEBUG:teuthology.nuke:shortname: smithi040 2024-02-14T17:43:43.018 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-14T17:43:43.042 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-14 17:21:36.120489', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGGEsMPMdo9+fZ0LxSlJIJJ1uEU6n+mFAeBWGFiPUe78hyRZb+CkOxV1dCgGRRpp0TMsu/Wb4Hs+1UyfneBZvC8='} 2024-02-14T17:43:43.047 DEBUG:teuthology.nuke:shortname: smithi096 2024-02-14T17:43:43.047 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-14T17:43:43.076 INFO:teuthology.orchestra.console.smithi027:Power off 2024-02-14T17:43:43.076 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-14T17:43:43.087 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559809', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-14 17:21:36.121747', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOvWGh6+0Ube1UCR24AGzdGs1VW3rag1Uoq9le34niBLnU5EcbLmWqFjCYG1evsX65Wvo8AVtnmz+47XTUk5ypM='} 2024-02-14T17:43:43.100 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off 2024-02-14T17:43:43.100 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:43:43.122 INFO:teuthology.orchestra.console.smithi040:Power off 2024-02-14T17:43:43.122 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-14T17:43:43.132 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-02-14T17:43:43.144 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2024-02-14T17:43:43.145 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:43:43.161 INFO:teuthology.orchestra.console.smithi096:Power off 2024-02-14T17:43:43.162 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-14T17:43:43.171 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-02-14T17:43:43.185 DEBUG:teuthology.orchestra.console.smithi096:power off output: Chassis Power Control: Down/Off 2024-02-14T17:43:43.185 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:43:43.207 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-02-14T17:43:47.134 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:43:47.172 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:43:47.208 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:43:47.244 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-02-14T17:43:47.281 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-02-14T17:43:47.321 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-02-14T17:43:51.249 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:43:51.282 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:43:51.322 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:43:51.360 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off 2024-02-14T17:43:51.391 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-14T17:43:51.430 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-02-14T17:43:51.461 INFO:teuthology.orchestra.console.smithi027:Power off completed 2024-02-14T17:43:51.491 INFO:teuthology.orchestra.console.smithi040:Power off completed 2024-02-14T17:43:55.435 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:43:55.544 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is off 2024-02-14T17:43:55.645 INFO:teuthology.orchestra.console.smithi096:Power off completed 2024-02-14T17:43:55.750 INFO:teuthology.run:Summary data: description: orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} duration: 714.0015885829926 failure_reason: '"2024-02-14T17:39:16.096978+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_gabrioux@teuthology success: false 2024-02-14T17:43:55.751 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-14T17:43:55.835 INFO:teuthology.run:FAIL