2024-03-26T04:27:35.797 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T04:27:35.797 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028 --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} --name teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028/orig.config.yaml 2024-03-26T04:27:35.825 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T04:27:35.936 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028 branch: squid description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7622028' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 1822 sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a sleep_before_teardown: 0 subset: 53/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDBkrJvXZfRGk5fUXdlzv0HcvGm4QDe2mWwho5eKhM5SCQUzLQRsV9kS7aEF3FbKkYTpqhBhMBo9YfcN+AST1ug= smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJe7DArtM43IOtvCVGqhbM4xapv46UVf0TwxMzz1WtnDgCd0pCxvOFxBNsLc46+q8QwqDPdzseALl9AlDiP1IU= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFWqv+3na21AEw5KjMzBAXaEMz65jfbvuqS2maLaV/dTRixKIMWs8HGEcX0o8QeKdxMJ8dSdKvC2FTyWLk6naBg= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_21:08:11 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T04:27:35.936 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa; will attempt to use it 2024-03-26T04:27:35.937 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa/tasks 2024-03-26T04:27:35.937 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T04:27:35.939 INFO:teuthology.task.internal:Checking packages... 2024-03-26T04:27:35.963 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5e70124b46c2a671efcaae4690cd9ea72670710a' 2024-03-26T04:27:35.963 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T04:27:35.963 INFO:teuthology.packaging:ref: None 2024-03-26T04:27:35.963 INFO:teuthology.packaging:tag: None 2024-03-26T04:27:35.963 INFO:teuthology.packaging:branch: squid 2024-03-26T04:27:35.963 INFO:teuthology.packaging:sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:27:35.963 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-26T04:27:36.115 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1863.g9f04c91b 2024-03-26T04:27:36.116 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T04:27:36.122 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T04:27:36.122 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T04:27:36.136 INFO:teuthology.task.internal:Saving configuration 2024-03-26T04:27:36.146 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T04:27:36.154 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:27:36.176 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:22:39.310990', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDBkrJvXZfRGk5fUXdlzv0HcvGm4QDe2mWwho5eKhM5SCQUzLQRsV9kS7aEF3FbKkYTpqhBhMBo9YfcN+AST1ug='} 2024-03-26T04:27:36.194 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:22:39.312576', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJe7DArtM43IOtvCVGqhbM4xapv46UVf0TwxMzz1WtnDgCd0pCxvOFxBNsLc46+q8QwqDPdzseALl9AlDiP1IU='} 2024-03-26T04:27:36.216 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:22:39.314039', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFWqv+3na21AEw5KjMzBAXaEMz65jfbvuqS2maLaV/dTRixKIMWs8HGEcX0o8QeKdxMJ8dSdKvC2FTyWLk6naBg='} 2024-03-26T04:27:36.216 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T04:27:36.224 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T04:27:36.224 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-26T04:27:36.224 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-26T04:27:36.224 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T04:27:36.323 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdb02710f70>, signals=[15]) 2024-03-26T04:27:36.324 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T04:27:36.330 INFO:teuthology.task.internal:Opening connections... 2024-03-26T04:27:36.330 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T04:27:36.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:27:36.439 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-03-26T04:27:36.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:27:36.527 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-03-26T04:27:36.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:27:36.615 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T04:27:36.623 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-03-26T04:27:36.641 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-03-26T04:27:36.641 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-03-26T04:27:36.699 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-03-26T04:27:36.699 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-03-26T04:27:36.699 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-03-26T04:27:36.700 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-03-26T04:27:36.700 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-03-26T04:27:36.700 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-03-26T04:27:36.700 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T04:27:36.700 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-03-26T04:27:36.700 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-03-26T04:27:36.700 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T04:27:36.700 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-03-26T04:27:36.700 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T04:27:36.700 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T04:27:36.700 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T04:27:36.701 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-03-26T04:27:36.723 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-03-26T04:27:36.740 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-03-26T04:27:36.741 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-03-26T04:27:36.798 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-03-26T04:27:36.798 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-03-26T04:27:36.798 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-03-26T04:27:36.799 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-03-26T04:27:36.799 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-03-26T04:27:36.799 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-03-26T04:27:36.799 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T04:27:36.799 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-03-26T04:27:36.799 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-03-26T04:27:36.799 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T04:27:36.799 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-03-26T04:27:36.799 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T04:27:36.799 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T04:27:36.799 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T04:27:36.800 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-03-26T04:27:36.823 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-03-26T04:27:36.840 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-03-26T04:27:36.840 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-03-26T04:27:36.897 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-03-26T04:27:36.897 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-03-26T04:27:36.897 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-03-26T04:27:36.897 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-03-26T04:27:36.898 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-03-26T04:27:36.898 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-03-26T04:27:36.898 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T04:27:36.898 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-03-26T04:27:36.898 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-03-26T04:27:36.898 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T04:27:36.898 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-03-26T04:27:36.898 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T04:27:36.898 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T04:27:36.898 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T04:27:36.898 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-03-26T04:27:36.914 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T04:27:36.922 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T04:27:36.929 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T04:27:36.929 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-03-26T04:27:36.932 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-03-26T04:27:36.934 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-03-26T04:27:36.954 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T04:27:36.962 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T04:27:36.962 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-03-26T04:27:36.989 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-03-26T04:27:36.992 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-03-26T04:27:37.092 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T04:27:37.170 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T04:27:37.185 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T04:27:37.186 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T04:27:37.186 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T04:27:37.186 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T04:27:37.187 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T04:27:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:27:37.203 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-03-26T04:27:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:27:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:27:37.205 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-03-26T04:27:37.205 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-03-26T04:27:37.261 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-361.el9.x86_64 2024-03-26T04:27:37.261 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-361.el9.x86_64 2024-03-26T04:27:37.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-03-26T04:27:37.263 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-361.el9.x86_64 2024-03-26T04:27:37.263 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-361.el9.x86_64 2024-03-26T04:27:37.263 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.14.0-361.el9.x86_64 2024-03-26T04:27:37.263 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum install -y kernel 2024-03-26T04:27:37.264 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.14.0-361.el9.x86_64 2024-03-26T04:27:37.264 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-03-26T04:27:39.394 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-26T04:27:39.511 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-26T04:27:39.594 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-26T04:27:40.986 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-03-26T04:27:40.987 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-26T04:27:40.987 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-26T04:27:41.869 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-26T04:27:42.993 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.0 MB 00:01 2024-03-26T04:27:45.145 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 8.7 MB/s | 19 MB 00:02 2024-03-26T04:27:45.178 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-26T04:27:45.778 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-03-26T04:27:49.271 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-26T04:27:49.436 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 6.1 MB 00:00 2024-03-26T04:27:50.905 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-26T04:27:50.921 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-03-26T04:27:52.949 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-26T04:27:53.007 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:01 2024-03-26T04:27:57.905 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-26T04:27:58.075 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:01 ago on Tue 26 Mar 2024 04:27:57 AM UTC. 2024-03-26T04:27:58.190 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-26T04:28:00.153 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T04:28:00.153 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T04:28:00.250 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-26T04:28:00.260 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-26T04:28:00.261 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-03-26T04:28:00.261 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-26T04:28:00.261 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-26T04:28:00.261 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T04:28:00.261 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-26T04:28:00.261 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T04:28:00.261 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T04:28:00.261 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T04:28:00.261 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-26T04:28:00.261 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-26T04:28:00.261 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-26T04:28:00.261 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-03-26T04:28:00.261 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-26T04:28:00.262 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 76 M 2024-03-26T04:28:00.262 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 123 M 2024-03-26T04:28:00.262 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-26T04:28:00.473 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 868 kB/s | 66 kB 00:00 2024-03-26T04:28:00.589 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T04:28:00.589 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T04:28:00.680 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-26T04:28:00.686 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-26T04:28:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-03-26T04:28:00.686 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-26T04:28:00.686 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-26T04:28:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T04:28:00.686 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-03-26T04:28:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T04:28:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T04:28:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T04:28:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:28:00.687 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-26T04:28:00.687 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-26T04:28:00.687 INFO:teuthology.orchestra.run.smithi038.stdout:Install 4 Packages 2024-03-26T04:28:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:28:00.687 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 76 M 2024-03-26T04:28:00.688 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 123 M 2024-03-26T04:28:00.688 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-26T04:28:00.853 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-26T04:28:00.975 INFO:teuthology.orchestra.run.smithi038.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 407 kB/s | 66 kB 00:00 2024-03-26T04:28:01.398 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 15 MB/s | 15 MB 00:01 2024-03-26T04:28:02.157 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 19 MB/s | 33 MB 00:01 2024-03-26T04:28:02.492 INFO:teuthology.orchestra.run.smithi038.stdout:(2/4): kernel-modules-core-5.14.0-432.el9.x86_6 18 MB/s | 27 MB 00:01 2024-03-26T04:28:02.624 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 13 MB/s | 27 MB 00:02 2024-03-26T04:28:02.624 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:28:02.624 INFO:teuthology.orchestra.run.smithi114.stdout:Total 32 MB/s | 76 MB 00:02 2024-03-26T04:28:02.658 INFO:teuthology.orchestra.run.smithi038.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 8.1 MB/s | 15 MB 00:01 2024-03-26T04:28:03.101 INFO:teuthology.orchestra.run.smithi038.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 15 MB/s | 33 MB 00:02 2024-03-26T04:28:03.101 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:28:03.101 INFO:teuthology.orchestra.run.smithi038.stdout:Total 31 MB/s | 76 MB 00:02 2024-03-26T04:28:03.291 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-26T04:28:03.636 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-26T04:28:03.637 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-26T04:28:03.794 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-26T04:28:04.227 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-26T04:28:04.228 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-26T04:28:04.630 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-26T04:28:04.631 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-26T04:28:04.776 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-26T04:28:05.279 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-26T04:28:05.280 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-26T04:28:06.244 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 16 kB 00:00 2024-03-26T04:28:07.073 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-26T04:28:07.671 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:28:07.770 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-26T04:28:07.773 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:28:07.882 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-26T04:28:08.503 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:28:08.594 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:28:10.482 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:28:10.608 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:28:11.467 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:28:11.609 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:28:12.670 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-26T04:28:14.843 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T04:28:14.844 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T04:28:14.939 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-26T04:28:14.951 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-26T04:28:14.951 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-03-26T04:28:14.951 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-26T04:28:14.952 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-26T04:28:14.952 INFO:teuthology.orchestra.run.smithi177.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T04:28:14.952 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-26T04:28:14.952 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T04:28:14.952 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T04:28:14.952 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T04:28:14.952 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:28:14.952 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-26T04:28:14.952 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-26T04:28:14.952 INFO:teuthology.orchestra.run.smithi177.stdout:Install 4 Packages 2024-03-26T04:28:14.952 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:28:14.953 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 76 M 2024-03-26T04:28:14.953 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 123 M 2024-03-26T04:28:14.953 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-26T04:28:15.154 INFO:teuthology.orchestra.run.smithi177.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 778 kB/s | 66 kB 00:00 2024-03-26T04:28:15.313 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:28:15.350 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:15.713 INFO:teuthology.orchestra.run.smithi177.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-26T04:28:16.221 INFO:teuthology.orchestra.run.smithi177.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 29 MB/s | 33 MB 00:01 2024-03-26T04:28:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:28:16.718 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:16.805 INFO:teuthology.orchestra.run.smithi177.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 17 MB/s | 27 MB 00:01 2024-03-26T04:28:16.805 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:28:16.805 INFO:teuthology.orchestra.run.smithi177.stdout:Total 41 MB/s | 76 MB 00:01 2024-03-26T04:28:17.468 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-26T04:28:17.826 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-26T04:28:17.827 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-26T04:28:18.840 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-26T04:28:18.840 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-26T04:28:20.090 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-26T04:28:21.812 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:21.912 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:28:21.997 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:28:24.764 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:28:24.900 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:28:29.592 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:28:29.625 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:34.382 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:49.164 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:49.164 INFO:teuthology.orchestra.run.smithi114.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T04:28:49.164 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-26T04:28:49.209 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:49.562 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:49.562 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:28:49.562 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:28:49.562 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:28:49.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:49.941 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-26T04:28:49.941 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-26T04:28:49.941 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T04:28:49.941 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T04:28:49.941 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T04:28:49.941 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T04:28:49.941 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-26T04:28:49.941 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-26T04:28:50.274 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:28:50.375 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:28:50.376 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:28:50.376 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T04:28:50.376 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T04:28:50.376 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T04:28:50.376 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T04:28:50.376 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T04:28:50.376 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:28:50.456 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:28:50.456 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:28:50.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-03-26T04:28:50.490 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-03-26T04:28:50.490 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-03-26T04:28:50.490 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-03-26T04:28:50.490 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 68.el9 2024-03-26T04:28:50.490 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-03-26T04:28:50.490 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T04:28:50.490 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8264254 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T04:28:50.491 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-03-26T04:28:50.492 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-26T04:28:50.492 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T04:28:50.492 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T04:28:50.492 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 5.14.0-432.el9.x86_64 2024-03-26T04:28:50.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T04:28:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-03-26T04:28:52.439 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T04:28:52.474 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-03-26T04:28:52.478 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-03-26T04:28:52.499 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.6nNEHy5n5h 2024-03-26T04:28:52.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6nNEHy5n5h 2024-03-26T04:28:52.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.6nNEHy5n5h 2024-03-26T04:28:52.692 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:52.692 INFO:teuthology.orchestra.run.smithi038.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T04:28:52.692 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:28:52.699 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.6nNEHy5n5h is 6KB 2024-03-26T04:28:52.738 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:52.747 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.6nNEHy5n5h 2024-03-26T04:28:52.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T04:28:52.826 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T04:28:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T04:28:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T04:28:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T04:28:52.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T04:28:53.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-03-26T04:28:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:28:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:28:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:28:53.492 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:53.492 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:28:53.492 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-26T04:28:53.492 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T04:28:53.492 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T04:28:53.492 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T04:28:53.492 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T04:28:53.492 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:28:53.492 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-26T04:28:53.819 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:28:53.924 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:28:53.924 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:28:53.924 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T04:28:53.924 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T04:28:53.924 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T04:28:53.924 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T04:28:53.924 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T04:28:53.924 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:28:54.007 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:28:54.007 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:28:54.007 DEBUG:teuthology.orchestra.run.smithi038:> sudo rpm -qi grub2-tools 2024-03-26T04:28:54.053 INFO:teuthology.orchestra.run.smithi038.stdout:Name : grub2-tools 2024-03-26T04:28:54.053 INFO:teuthology.orchestra.run.smithi038.stdout:Epoch : 1 2024-03-26T04:28:54.053 INFO:teuthology.orchestra.run.smithi038.stdout:Version : 2.06 2024-03-26T04:28:54.053 INFO:teuthology.orchestra.run.smithi038.stdout:Release : 68.el9 2024-03-26T04:28:54.053 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: x86_64 2024-03-26T04:28:54.053 INFO:teuthology.orchestra.run.smithi038.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T04:28:54.053 INFO:teuthology.orchestra.run.smithi038.stdout:Group : Unspecified 2024-03-26T04:28:54.053 INFO:teuthology.orchestra.run.smithi038.stdout:Size : 8264254 2024-03-26T04:28:54.053 INFO:teuthology.orchestra.run.smithi038.stdout:License : GPLv3+ 2024-03-26T04:28:54.054 INFO:teuthology.orchestra.run.smithi038.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T04:28:54.054 INFO:teuthology.orchestra.run.smithi038.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T04:28:54.054 INFO:teuthology.orchestra.run.smithi038.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T04:28:54.054 INFO:teuthology.orchestra.run.smithi038.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T04:28:54.054 INFO:teuthology.orchestra.run.smithi038.stdout:Packager : builder@centos.org 2024-03-26T04:28:54.054 INFO:teuthology.orchestra.run.smithi038.stdout:Vendor : CentOS 2024-03-26T04:28:54.054 INFO:teuthology.orchestra.run.smithi038.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T04:28:54.054 INFO:teuthology.orchestra.run.smithi038.stdout:Summary : Support tools for GRUB. 2024-03-26T04:28:54.054 INFO:teuthology.orchestra.run.smithi038.stdout:Description : 2024-03-26T04:28:54.054 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:28:54.054 INFO:teuthology.orchestra.run.smithi038.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T04:28:54.054 INFO:teuthology.orchestra.run.smithi038.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T04:28:54.054 INFO:teuthology.orchestra.run.smithi038.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T04:28:54.055 INFO:teuthology.orchestra.run.smithi038.stdout:hardware devices. 2024-03-26T04:28:54.055 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:28:54.055 INFO:teuthology.orchestra.run.smithi038.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T04:28:54.055 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T04:28:54.055 INFO:teuthology.task.kernel:Updating grub on smithi038 to boot 5.14.0-432.el9.x86_64 2024-03-26T04:28:54.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T04:28:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2024-03-26T04:28:56.059 INFO:teuthology.orchestra.run.smithi038.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T04:28:56.093 INFO:teuthology.orchestra.run.smithi038.stderr:done 2024-03-26T04:28:56.095 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-03-26T04:28:56.110 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.9fAldmNICI 2024-03-26T04:28:56.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9fAldmNICI 2024-03-26T04:28:56.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.9fAldmNICI 2024-03-26T04:28:56.321 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.9fAldmNICI is 6KB 2024-03-26T04:28:56.369 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.9fAldmNICI 2024-03-26T04:28:56.384 DEBUG:teuthology.orchestra.run.smithi038:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T04:28:56.452 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T04:28:56.453 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T04:28:56.453 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T04:28:56.453 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T04:28:56.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T04:28:56.595 DEBUG:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2024-03-26T04:29:03.635 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:03.635 INFO:teuthology.orchestra.run.smithi177.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T04:29:03.635 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:29:03.679 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:04.029 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:04.029 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:29:04.029 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:29:04.029 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:29:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:29:04.404 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-26T04:29:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T04:29:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T04:29:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T04:29:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T04:29:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:29:04.404 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-26T04:29:04.695 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:29:04.802 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:29:04.802 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:29:04.802 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T04:29:04.802 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T04:29:04.802 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T04:29:04.802 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-26T04:29:04.803 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T04:29:04.803 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:29:04.883 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:29:04.883 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:29:04.884 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -qi grub2-tools 2024-03-26T04:29:04.918 INFO:teuthology.orchestra.run.smithi177.stdout:Name : grub2-tools 2024-03-26T04:29:04.918 INFO:teuthology.orchestra.run.smithi177.stdout:Epoch : 1 2024-03-26T04:29:04.918 INFO:teuthology.orchestra.run.smithi177.stdout:Version : 2.06 2024-03-26T04:29:04.919 INFO:teuthology.orchestra.run.smithi177.stdout:Release : 68.el9 2024-03-26T04:29:04.919 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: x86_64 2024-03-26T04:29:04.919 INFO:teuthology.orchestra.run.smithi177.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T04:29:04.919 INFO:teuthology.orchestra.run.smithi177.stdout:Group : Unspecified 2024-03-26T04:29:04.919 INFO:teuthology.orchestra.run.smithi177.stdout:Size : 8264254 2024-03-26T04:29:04.919 INFO:teuthology.orchestra.run.smithi177.stdout:License : GPLv3+ 2024-03-26T04:29:04.919 INFO:teuthology.orchestra.run.smithi177.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T04:29:04.919 INFO:teuthology.orchestra.run.smithi177.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T04:29:04.919 INFO:teuthology.orchestra.run.smithi177.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T04:29:04.919 INFO:teuthology.orchestra.run.smithi177.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T04:29:04.919 INFO:teuthology.orchestra.run.smithi177.stdout:Packager : builder@centos.org 2024-03-26T04:29:04.919 INFO:teuthology.orchestra.run.smithi177.stdout:Vendor : CentOS 2024-03-26T04:29:04.919 INFO:teuthology.orchestra.run.smithi177.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T04:29:04.920 INFO:teuthology.orchestra.run.smithi177.stdout:Summary : Support tools for GRUB. 2024-03-26T04:29:04.920 INFO:teuthology.orchestra.run.smithi177.stdout:Description : 2024-03-26T04:29:04.920 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:29:04.920 INFO:teuthology.orchestra.run.smithi177.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T04:29:04.920 INFO:teuthology.orchestra.run.smithi177.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T04:29:04.920 INFO:teuthology.orchestra.run.smithi177.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T04:29:04.920 INFO:teuthology.orchestra.run.smithi177.stdout:hardware devices. 2024-03-26T04:29:04.920 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:29:04.920 INFO:teuthology.orchestra.run.smithi177.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T04:29:04.921 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T04:29:04.921 INFO:teuthology.task.kernel:Updating grub on smithi177 to boot 5.14.0-432.el9.x86_64 2024-03-26T04:29:04.921 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T04:29:05.566 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-03-26T04:29:06.928 INFO:teuthology.orchestra.run.smithi177.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T04:29:06.955 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-03-26T04:29:06.957 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-03-26T04:29:06.972 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.AvAk1PZGRh 2024-03-26T04:29:06.972 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AvAk1PZGRh 2024-03-26T04:29:07.041 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.AvAk1PZGRh 2024-03-26T04:29:07.174 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.AvAk1PZGRh is 6KB 2024-03-26T04:29:07.221 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.AvAk1PZGRh 2024-03-26T04:29:07.235 DEBUG:teuthology.orchestra.run.smithi177:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T04:29:07.307 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T04:29:07.307 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T04:29:07.307 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T04:29:07.307 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T04:29:07.308 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T04:29:07.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-03-26T04:29:23.027 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T04:29:23.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-26T04:29:23.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:29:26.598 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T04:29:26.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-26T04:29:26.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:29:37.463 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-26T04:29:37.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-26T04:29:37.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:29:41.535 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-03-26T04:29:45.125 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2024-03-26T04:29:50.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-26T04:29:50.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:29:54.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-26T04:29:54.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:29:55.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-26T04:30:04.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-26T04:30:04.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:30:07.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-26T04:30:19.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-26T04:30:19.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:30:38.367 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-26T04:30:50.557 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T04:30:53.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-26T04:30:53.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:30:53.758 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-03-26T04:30:54.133 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T04:30:54.173 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-26T04:30:54.173 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-432.el9.x86_64"... 2024-03-26T04:30:54.173 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-03-26T04:30:54.187 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-432.el9.x86_64 2024-03-26T04:30:54.187 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T04:30:54.187 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T04:30:54.187 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T04:30:55.188 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-26T04:30:55.189 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T04:30:55.254 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-03-26T04:30:55.324 DEBUG:teuthology.parallel:result is None 2024-03-26T04:31:02.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-26T04:31:02.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:31:02.875 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-03-26T04:31:03.265 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-26T04:31:03.266 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T04:31:03.266 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-03-26T04:31:03.281 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-432.el9.x86_64 2024-03-26T04:31:03.281 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T04:31:03.281 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T04:31:03.281 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T04:31:04.282 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T04:31:04.282 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T04:31:04.349 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2024-03-26T04:31:04.383 DEBUG:teuthology.parallel:result is None 2024-03-26T04:31:06.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-26T04:31:06.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:31:06.478 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-03-26T04:31:06.888 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-26T04:31:06.889 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T04:31:06.889 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-03-26T04:31:06.903 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-432.el9.x86_64 2024-03-26T04:31:06.903 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T04:31:06.904 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T04:31:06.904 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T04:31:07.905 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T04:31:07.905 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T04:31:07.984 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2024-03-26T04:31:08.020 DEBUG:teuthology.parallel:result is None 2024-03-26T04:31:08.020 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T04:31:08.029 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T04:31:08.029 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T04:31:08.031 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T04:31:08.034 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T04:31:08.052 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T04:31:08.068 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T04:31:08.110 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T04:31:08.111 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T04:31:08.113 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T04:31:08.115 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T04:31:08.211 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T04:31:08.218 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T04:31:08.218 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T04:31:08.221 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T04:31:08.223 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T04:31:08.256 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:31:08.260 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:31:08.260 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:31:08.270 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:31:08.274 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:31:08.275 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:31:08.277 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T04:31:08.284 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T04:31:08.284 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T04:31:08.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T04:31:08.321 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T04:31:08.352 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T04:31:08.362 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T04:31:08.362 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T04:31:08.389 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T04:31:08.391 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T04:31:08.409 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:31:08.485 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:31:08.541 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:31:08.620 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:31:08.667 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:31:08.667 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T04:31:08.736 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:31:08.782 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:31:08.835 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:31:08.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:31:08.964 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-26T04:31:08.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T04:31:09.033 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:31:09.078 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:31:09.136 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:31:09.221 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:31:09.267 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-26T04:31:09.267 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T04:31:09.336 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-03-26T04:31:09.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-03-26T04:31:09.340 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-03-26T04:31:09.384 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:31:09.385 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:31:09.430 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:31:09.936 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T04:31:09.943 INFO:teuthology.task.internal:Starting timer... 2024-03-26T04:31:09.943 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T04:31:09.989 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T04:31:10.019 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-26T04:31:10.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-03-26T04:31:10.076 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-03-26T04:31:10.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-03-26T04:31:10.132 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-03-26T04:31:10.134 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-03-26T04:31:10.211 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-03-26T04:31:10.213 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T04:31:10.213 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-03-26T04:31:10.234 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-03-26T04:31:10.234 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-03-26T04:31:10.269 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-03-26T04:31:10.269 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-03-26T04:31:10.297 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-03-26T04:31:10.298 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-03-26T04:31:10.298 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:31:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:31:10.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:31:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:31:10.355 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:31:10.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:31:10.384 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T04:31:10.384 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-03-26T04:31:10.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-03-26T04:31:10.455 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-03-26T04:31:10.492 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T04:31:10.501 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T04:31:10.518 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T04:31:10.519 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi114.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T04:35:47.977 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-03-26T04:35:47.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-26T04:35:47.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:35:48.071 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-03-26T04:35:48.162 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-26T04:35:48.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-26T04:35:48.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:35:48.250 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-03-26T04:35:48.330 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-26T04:35:48.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-26T04:35:48.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:35:48.420 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-03-26T04:35:48.499 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-26T04:35:48.500 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T04:35:48.510 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T04:35:48.510 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:35:48.511 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:35:48.513 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:35:48.514 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:35:48.516 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:35:48.516 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:35:48.546 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:35:48.547 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:35:48.564 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:35:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:35:48.575 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:35:48.593 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:35:48.595 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-03-26T04:35:48.595 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-03-26T04:35:48.609 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-03-26T04:35:48.610 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-03-26T04:35:48.624 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-03-26T04:35:48.626 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:35:48.626 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:35:48.638 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-03-26T04:35:48.642 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:35:48.643 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:35:48.656 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:35:48.673 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:35:48.699 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:35:48.701 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:35:48.702 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:35:48.702 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-03-26T04:35:48.702 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:35:48.702 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:35:48.702 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:35:48.702 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:35:48.704 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:35:48.704 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-03-26T04:35:48.704 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:35:48.704 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:35:48.704 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:35:48.704 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:35:48.728 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:35:48.731 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:35:48.731 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-03-26T04:35:48.731 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:35:48.732 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:35:48.732 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:35:48.732 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:35:48.733 INFO:teuthology.run_tasks:Running task pexec... 2024-03-26T04:35:48.742 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-26T04:35:48.742 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T04:35:48.742 DEBUG:teuthology.orchestra.run.smithi038:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T04:35:48.742 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi114.front.sepia.ceph.com 2024-03-26T04:35:48.743 DEBUG:teuthology.orchestra.run.smithi114:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T04:35:48.743 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi177.front.sepia.ceph.com 2024-03-26T04:35:48.743 DEBUG:teuthology.orchestra.run.smithi177:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T04:35:49.317 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:13 ago on Tue 26 Mar 2024 04:34:36 AM UTC. 2024-03-26T04:35:49.330 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:07 ago on Tue 26 Mar 2024 04:34:42 AM UTC. 2024-03-26T04:35:49.350 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:11 ago on Tue 26 Mar 2024 04:34:38 AM UTC. 2024-03-26T04:35:49.462 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-26T04:35:49.470 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-26T04:35:49.470 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-03-26T04:35:49.471 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-26T04:35:49.471 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-26T04:35:49.471 INFO:teuthology.orchestra.run.smithi114.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T04:35:49.471 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-26T04:35:49.471 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-26T04:35:49.471 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-26T04:35:49.471 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-26T04:35:49.471 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-26T04:35:49.471 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 3.1 M 2024-03-26T04:35:49.472 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 9.8 M 2024-03-26T04:35:49.472 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-26T04:35:49.482 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-26T04:35:49.484 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-26T04:35:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-03-26T04:35:49.484 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-26T04:35:49.484 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-26T04:35:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T04:35:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:35:49.484 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-26T04:35:49.484 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-26T04:35:49.484 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-03-26T04:35:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:35:49.485 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.1 M 2024-03-26T04:35:49.485 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 9.8 M 2024-03-26T04:35:49.485 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-26T04:35:49.497 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-26T04:35:49.511 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-26T04:35:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-03-26T04:35:49.511 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-26T04:35:49.511 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-26T04:35:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T04:35:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:35:49.511 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-26T04:35:49.511 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-26T04:35:49.511 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-26T04:35:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:35:49.511 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 3.1 M 2024-03-26T04:35:49.512 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 9.8 M 2024-03-26T04:35:49.512 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-26T04:35:49.838 INFO:teuthology.orchestra.run.smithi114.stdout:runc-1.1.12-2.el9.x86_64.rpm 14 MB/s | 3.1 MB 00:00 2024-03-26T04:35:49.838 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:35:49.838 INFO:teuthology.orchestra.run.smithi114.stdout:Total 8.5 MB/s | 3.1 MB 00:00 2024-03-26T04:35:49.874 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-26T04:35:49.880 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-26T04:35:49.880 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-26T04:35:49.960 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-26T04:35:49.960 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-26T04:35:50.137 INFO:teuthology.orchestra.run.smithi177.stdout:runc-1.1.12-2.el9.x86_64.rpm 9.1 MB/s | 3.1 MB 00:00 2024-03-26T04:35:50.137 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:35:50.137 INFO:teuthology.orchestra.run.smithi177.stdout:Total 5.0 MB/s | 3.1 MB 00:00 2024-03-26T04:35:50.170 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-26T04:35:50.173 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-26T04:35:50.179 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-26T04:35:50.179 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-26T04:35:50.258 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-26T04:35:50.259 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-26T04:35:50.266 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:35:50.479 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-26T04:35:50.581 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:35:50.696 INFO:teuthology.orchestra.run.smithi038.stdout:runc-1.1.12-2.el9.x86_64.rpm 2.9 MB/s | 3.1 MB 00:01 2024-03-26T04:35:50.696 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:35:50.697 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.6 MB/s | 3.1 MB 00:01 2024-03-26T04:35:50.736 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-26T04:35:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-26T04:35:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-26T04:35:50.826 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-26T04:35:50.827 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-26T04:35:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:35:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-26T04:35:51.121 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:35:51.121 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-26T04:35:51.121 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-26T04:35:51.122 INFO:teuthology.orchestra.run.smithi114.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T04:35:51.122 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-26T04:35:51.122 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-26T04:35:51.160 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:35:51.182 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:35:51.369 DEBUG:teuthology.parallel:result is None 2024-03-26T04:35:51.496 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:35:51.496 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:35:51.496 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-26T04:35:51.496 INFO:teuthology.orchestra.run.smithi177.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T04:35:51.496 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:35:51.496 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-26T04:35:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:35:51.737 DEBUG:teuthology.parallel:result is None 2024-03-26T04:35:52.030 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:35:52.030 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:35:52.031 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-26T04:35:52.031 INFO:teuthology.orchestra.run.smithi038.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T04:35:52.031 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:35:52.031 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-26T04:35:52.311 DEBUG:teuthology.parallel:result is None 2024-03-26T04:35:52.311 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-26T04:35:52.322 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-26T04:35:52.322 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:35:52.322 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:35:52.337 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:35:52.337 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-03-26T04:35:52.392 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:35:52.392 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:35:52.392 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T04:35:52.392 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:35:52.392 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:35:52.392 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-26 04:35:46.888791699 +0000 2024-03-26T04:35:52.392 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-26 04:35:11.626511610 +0000 2024-03-26T04:35:52.393 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-26 04:35:11.626511610 +0000 2024-03-26T04:35:52.393 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-26 04:35:11.626511610 +0000 2024-03-26T04:35:52.393 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:35:52.457 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-26T04:35:52.457 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-26T04:35:52.457 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.00027307 s, 1.9 MB/s 2024-03-26T04:35:52.458 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:35:52.514 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-03-26T04:35:52.569 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:35:52.569 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:35:52.569 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-26T04:35:52.569 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:35:52.569 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:35:52.569 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-26 04:35:46.888791699 +0000 2024-03-26T04:35:52.569 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-26 04:35:11.977504444 +0000 2024-03-26T04:35:52.569 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-26 04:35:11.977504444 +0000 2024-03-26T04:35:52.569 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-26 04:35:11.977504444 +0000 2024-03-26T04:35:52.569 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:35:52.633 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-26T04:35:52.633 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-26T04:35:52.633 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000277264 s, 1.8 MB/s 2024-03-26T04:35:52.634 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:35:52.692 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-03-26T04:35:52.746 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:35:52.746 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:35:52.746 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-26T04:35:52.746 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:35:52.746 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:35:52.747 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-26 04:35:46.888791699 +0000 2024-03-26T04:35:52.747 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-26 04:35:12.337497095 +0000 2024-03-26T04:35:52.747 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-26 04:35:12.337497095 +0000 2024-03-26T04:35:52.747 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-26 04:35:12.337497095 +0000 2024-03-26T04:35:52.747 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:35:52.812 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-26T04:35:52.812 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-26T04:35:52.812 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000265723 s, 1.9 MB/s 2024-03-26T04:35:52.813 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:35:52.869 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-03-26T04:35:52.923 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:35:52.924 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:35:52.924 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-26T04:35:52.924 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:35:52.924 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:35:52.924 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-26 04:35:46.888791699 +0000 2024-03-26T04:35:52.924 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-26 04:35:12.692489847 +0000 2024-03-26T04:35:52.924 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-26 04:35:12.692489847 +0000 2024-03-26T04:35:52.924 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-26 04:35:12.692489847 +0000 2024-03-26T04:35:52.924 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:35:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-26T04:35:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-26T04:35:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000273372 s, 1.9 MB/s 2024-03-26T04:35:53.010 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:35:53.025 DEBUG:teuthology.orchestra.run.smithi038:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-26T04:35:53.160 INFO:teuthology.orchestra.run.smithi038.stdout:loop 2024-03-26T04:35:53.162 INFO:tasks.nvme_loop:Connecting nvme_loop smithi038:/dev/vg_nvme/lv_1... 2024-03-26T04:35:53.162 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-26T04:35:53.199 INFO:teuthology.orchestra.run.smithi038.stdout:1 2024-03-26T04:35:53.236 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/vg_nvme/lv_11 2024-03-26T04:35:53.275 INFO:tasks.nvme_loop:Connecting nvme_loop smithi038:/dev/vg_nvme/lv_2... 2024-03-26T04:35:53.275 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-26T04:35:53.354 INFO:teuthology.orchestra.run.smithi038.stdout:1 2024-03-26T04:35:53.392 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/vg_nvme/lv_21 2024-03-26T04:35:53.421 INFO:tasks.nvme_loop:Connecting nvme_loop smithi038:/dev/vg_nvme/lv_3... 2024-03-26T04:35:53.422 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-26T04:35:53.501 INFO:teuthology.orchestra.run.smithi038.stdout:1 2024-03-26T04:35:53.540 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/vg_nvme/lv_31 2024-03-26T04:35:53.576 INFO:tasks.nvme_loop:Connecting nvme_loop smithi038:/dev/vg_nvme/lv_4... 2024-03-26T04:35:53.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-26T04:35:53.672 INFO:teuthology.orchestra.run.smithi038.stdout:1 2024-03-26T04:35:53.711 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/vg_nvme/lv_41 2024-03-26T04:35:53.752 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:35:53.752 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:35:53.767 DEBUG:teuthology.orchestra.run.smithi038:> sudo nvme list 2024-03-26T04:35:53.832 INFO:teuthology.orchestra.run.smithi038.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-26T04:35:53.832 INFO:teuthology.orchestra.run.smithi038.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-26T04:35:53.832 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme0n1 BTLE736103BB4P0KGN INTEL SSDPEDKE040T7 1 4.00 TB / 4.00 TB 512 B + 0 B QDV1013D 2024-03-26T04:35:53.832 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme1n1 42987afa66f2869e95ec Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:35:53.832 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n1 785c9bc245ec3afce867 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:35:53.832 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme3n1 6da9fb742065fdcf1a12 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:35:53.832 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme4n1 bd03baa47c07a6a493f6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:35:53.833 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T04:35:53.833 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:35:53.833 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/scratch_devs 2024-03-26T04:35:53.900 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-26T04:35:53.900 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:35:53.917 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:35:53.917 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-03-26T04:35:53.971 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:35:53.971 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:35:53.972 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T04:35:53.972 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:35:53.972 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:35:53.972 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-26 04:35:46.899759638 +0000 2024-03-26T04:35:53.972 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-26 04:35:01.900898158 +0000 2024-03-26T04:35:53.972 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-26 04:35:01.900898158 +0000 2024-03-26T04:35:53.972 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-26 04:35:01.900898158 +0000 2024-03-26T04:35:53.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:35:54.038 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-26T04:35:54.038 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-26T04:35:54.039 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000205437 s, 2.5 MB/s 2024-03-26T04:35:54.040 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:35:54.096 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-03-26T04:35:54.150 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:35:54.150 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:35:54.150 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-26T04:35:54.150 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:35:54.151 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:35:54.151 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-26 04:35:46.899759638 +0000 2024-03-26T04:35:54.151 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-26 04:35:02.258897055 +0000 2024-03-26T04:35:54.151 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-26 04:35:02.258897055 +0000 2024-03-26T04:35:54.151 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-26 04:35:02.258897055 +0000 2024-03-26T04:35:54.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:35:54.216 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-26T04:35:54.216 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-26T04:35:54.216 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000201756 s, 2.5 MB/s 2024-03-26T04:35:54.217 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:35:54.273 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-03-26T04:35:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:35:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:35:54.327 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-26T04:35:54.327 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:35:54.327 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:35:54.327 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-26 04:35:46.899759638 +0000 2024-03-26T04:35:54.327 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-26 04:35:02.598896009 +0000 2024-03-26T04:35:54.327 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-26 04:35:02.598896009 +0000 2024-03-26T04:35:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-26 04:35:02.598896009 +0000 2024-03-26T04:35:54.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:35:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-26T04:35:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-26T04:35:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000165195 s, 3.1 MB/s 2024-03-26T04:35:54.393 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:35:54.449 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-03-26T04:35:54.503 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:35:54.503 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:35:54.503 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-26T04:35:54.503 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:35:54.503 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:35:54.503 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-26 04:35:46.899759638 +0000 2024-03-26T04:35:54.504 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-26 04:35:02.957894904 +0000 2024-03-26T04:35:54.504 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-26 04:35:02.957894904 +0000 2024-03-26T04:35:54.504 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-26 04:35:02.957894904 +0000 2024-03-26T04:35:54.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:35:54.568 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-26T04:35:54.568 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-26T04:35:54.568 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.00016563 s, 3.1 MB/s 2024-03-26T04:35:54.569 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:35:54.624 DEBUG:teuthology.orchestra.run.smithi114:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-26T04:35:54.759 INFO:teuthology.orchestra.run.smithi114.stdout:loop 2024-03-26T04:35:54.760 INFO:tasks.nvme_loop:Connecting nvme_loop smithi114:/dev/vg_nvme/lv_1... 2024-03-26T04:35:54.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-26T04:35:54.798 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-26T04:35:54.835 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/vg_nvme/lv_11 2024-03-26T04:35:54.864 INFO:tasks.nvme_loop:Connecting nvme_loop smithi114:/dev/vg_nvme/lv_2... 2024-03-26T04:35:54.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-26T04:35:54.944 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-26T04:35:54.982 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/vg_nvme/lv_21 2024-03-26T04:35:55.010 INFO:tasks.nvme_loop:Connecting nvme_loop smithi114:/dev/vg_nvme/lv_3... 2024-03-26T04:35:55.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-26T04:35:55.089 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-26T04:35:55.126 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/vg_nvme/lv_31 2024-03-26T04:35:55.154 INFO:tasks.nvme_loop:Connecting nvme_loop smithi114:/dev/vg_nvme/lv_4... 2024-03-26T04:35:55.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-26T04:35:55.232 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-26T04:35:55.268 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/vg_nvme/lv_41 2024-03-26T04:35:55.303 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-26T04:35:55.304 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:35:55.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo nvme list 2024-03-26T04:35:55.424 INFO:teuthology.orchestra.run.smithi114.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-26T04:35:55.424 INFO:teuthology.orchestra.run.smithi114.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-26T04:35:55.425 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/nvme0n1 PHFT620400WB400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-26T04:35:55.425 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/nvme1n1 e51e45fb85e9164a05a8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:35:55.425 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/nvme2n1 9be6cd64173c051bffe5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:35:55.425 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/nvme3n1 8398787abebc4cfadfba Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:35:55.425 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/nvme4n1 7af3eda6431ec951488c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:35:55.426 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T04:35:55.426 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-26T04:35:55.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/scratch_devs 2024-03-26T04:35:55.492 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-26T04:35:55.492 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:35:55.511 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:35:55.511 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-03-26T04:35:55.570 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:35:55.570 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:35:55.570 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-26T04:35:55.570 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:35:55.570 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:35:55.570 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-26 04:35:46.900665452 +0000 2024-03-26T04:35:55.570 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-26 04:35:09.506729870 +0000 2024-03-26T04:35:55.570 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-26 04:35:09.506729870 +0000 2024-03-26T04:35:55.570 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-26 04:35:09.506729870 +0000 2024-03-26T04:35:55.571 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:35:55.641 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-26T04:35:55.641 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-26T04:35:55.641 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000306479 s, 1.7 MB/s 2024-03-26T04:35:55.642 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:35:55.701 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-03-26T04:35:55.759 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:35:55.759 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:35:55.759 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-26T04:35:55.759 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:35:55.759 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:35:55.759 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-26 04:35:46.900665452 +0000 2024-03-26T04:35:55.760 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-26 04:35:09.835720469 +0000 2024-03-26T04:35:55.760 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-26 04:35:09.835720469 +0000 2024-03-26T04:35:55.760 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-26 04:35:09.835720469 +0000 2024-03-26T04:35:55.760 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:35:55.829 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-26T04:35:55.829 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-26T04:35:55.829 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000201815 s, 2.5 MB/s 2024-03-26T04:35:55.830 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:35:55.888 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-03-26T04:35:55.947 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:35:55.947 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:35:55.947 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-26T04:35:55.947 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:35:55.947 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:35:55.947 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-26 04:35:46.900665452 +0000 2024-03-26T04:35:55.947 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-26 04:35:10.168710954 +0000 2024-03-26T04:35:55.947 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-26 04:35:10.168710954 +0000 2024-03-26T04:35:55.947 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-26 04:35:10.168710954 +0000 2024-03-26T04:35:55.948 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:35:56.014 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-26T04:35:56.014 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-26T04:35:56.014 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000169163 s, 3.0 MB/s 2024-03-26T04:35:56.016 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:35:56.073 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-03-26T04:35:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:35:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:35:56.129 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-26T04:35:56.129 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:35:56.130 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:35:56.130 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-26 04:35:46.900665452 +0000 2024-03-26T04:35:56.130 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-26 04:35:10.499701495 +0000 2024-03-26T04:35:56.130 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-26 04:35:10.499701495 +0000 2024-03-26T04:35:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-26 04:35:10.499701495 +0000 2024-03-26T04:35:56.130 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:35:56.196 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-26T04:35:56.196 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-26T04:35:56.196 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000196831 s, 2.6 MB/s 2024-03-26T04:35:56.197 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:35:56.254 DEBUG:teuthology.orchestra.run.smithi177:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-26T04:35:56.399 INFO:teuthology.orchestra.run.smithi177.stdout:loop 2024-03-26T04:35:56.401 INFO:tasks.nvme_loop:Connecting nvme_loop smithi177:/dev/vg_nvme/lv_1... 2024-03-26T04:35:56.401 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-26T04:35:56.438 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-03-26T04:35:56.475 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/vg_nvme/lv_11 2024-03-26T04:35:56.505 INFO:tasks.nvme_loop:Connecting nvme_loop smithi177:/dev/vg_nvme/lv_2... 2024-03-26T04:35:56.505 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-26T04:35:56.584 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-03-26T04:35:56.621 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/vg_nvme/lv_21 2024-03-26T04:35:56.649 INFO:tasks.nvme_loop:Connecting nvme_loop smithi177:/dev/vg_nvme/lv_3... 2024-03-26T04:35:56.650 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-26T04:35:56.728 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-03-26T04:35:56.765 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/vg_nvme/lv_31 2024-03-26T04:35:56.793 INFO:tasks.nvme_loop:Connecting nvme_loop smithi177:/dev/vg_nvme/lv_4... 2024-03-26T04:35:56.793 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-26T04:35:56.872 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-03-26T04:35:56.909 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/vg_nvme/lv_41 2024-03-26T04:35:56.937 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-26T04:35:56.937 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:35:56.992 DEBUG:teuthology.orchestra.run.smithi177:> sudo nvme list 2024-03-26T04:35:57.058 INFO:teuthology.orchestra.run.smithi177.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-26T04:35:57.058 INFO:teuthology.orchestra.run.smithi177.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-26T04:35:57.058 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/nvme0n1 CVFT6233007S400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-26T04:35:57.058 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/nvme1n1 a130a442367705bc3944 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:35:57.058 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/nvme2n1 51d01293df68df9b27cd Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:35:57.058 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/nvme3n1 c14b989b13a97bae137c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:35:57.058 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/nvme4n1 f7b8c90c5e43314389d7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:35:57.059 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T04:35:57.059 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-26T04:35:57.059 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/scratch_devs 2024-03-26T04:35:57.126 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T04:35:57.233 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'} 2024-03-26T04:35:57.233 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:35:57.233 INFO:tasks.cephadm:Cluster fsid is 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:35:57.234 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T04:35:57.234 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.38', 'mon.b': '172.21.15.114', 'mon.c': '172.21.15.177'} 2024-03-26T04:35:57.234 INFO:tasks.cephadm:First mon is mon.a on smithi038 2024-03-26T04:35:57.234 INFO:tasks.cephadm:First mgr is y 2024-03-26T04:35:57.234 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T04:35:57.234 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2024-03-26T04:35:57.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-03-26T04:35:57.299 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname $(hostname -s) 2024-03-26T04:35:57.325 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-26T04:35:57.326 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:35:57.603 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54690/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1775-g5e70124b', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1775.g5e70124b'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-25 18:23:16.641218', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-26T04:35:57.747 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 5e70124b46c2a671efcaae4690cd9ea72670710a from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:35:57.748 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm 2024-03-26T04:35:57.749 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T04:35:58.195 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 04:35 /home/ubuntu/cephtest/cephadm 2024-03-26T04:35:58.195 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T04:35:58.877 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 04:35 /home/ubuntu/cephtest/cephadm 2024-03-26T04:35:58.878 DEBUG:teuthology.orchestra.run.smithi177:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T04:35:59.494 INFO:teuthology.orchestra.run.smithi177.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 04:35 /home/ubuntu/cephtest/cephadm 2024-03-26T04:35:59.495 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T04:35:59.514 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T04:35:59.533 DEBUG:teuthology.orchestra.run.smithi177:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T04:35:59.564 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a on all hosts... 2024-03-26T04:35:59.564 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T04:35:59.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T04:35:59.575 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T04:36:00.359 INFO:teuthology.orchestra.run.smithi177.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:36:00.380 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:36:00.417 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:36:29.048 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-26T04:36:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:36:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T04:36:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-03-26T04:36:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T04:36:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-26T04:36:29.049 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-26T04:36:37.102 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-03-26T04:36:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:36:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T04:36:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "repo_digests": [ 2024-03-26T04:36:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T04:36:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-03-26T04:36:37.103 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-03-26T04:36:46.343 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-26T04:36:46.343 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:36:46.343 INFO:teuthology.orchestra.run.smithi038.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T04:36:46.343 INFO:teuthology.orchestra.run.smithi038.stdout: "repo_digests": [ 2024-03-26T04:36:46.343 INFO:teuthology.orchestra.run.smithi038.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T04:36:46.343 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-26T04:36:46.343 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-26T04:36:46.367 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2024-03-26T04:36:46.401 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-03-26T04:36:46.436 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph 2024-03-26T04:36:46.470 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2024-03-26T04:36:46.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-03-26T04:36:46.529 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /etc/ceph 2024-03-26T04:36:46.559 INFO:tasks.cephadm:Writing seed config... 2024-03-26T04:36:46.560 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T04:36:46.560 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T04:36:46.560 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-03-26T04:36:46.560 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-26T04:36:46.560 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T04:36:46.560 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T04:36:46.560 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T04:36:46.560 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T04:36:46.560 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T04:36:46.560 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T04:36:46.561 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:36:46.561 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T04:36:46.575 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 56caa506-eb2a-11ee-95d0-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-26T04:36:46.576 DEBUG:teuthology.orchestra.run.smithi038:mon.a> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T04:36:46.618 DEBUG:teuthology.orchestra.run.smithi038:mgr.y> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T04:36:46.660 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T04:36:46.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a -v bootstrap --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.38 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:36:46.856 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:36:46.856 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a', '-v', 'bootstrap', '--fsid', '56caa506-eb2a-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.38', '--skip-admin-label'] 2024-03-26T04:36:46.883 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:36:46.883 INFO:teuthology.orchestra.run.smithi038.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T04:36:46.884 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying podman|docker is present... 2024-03-26T04:36:46.906 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:36:46.906 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying lvm2 is present... 2024-03-26T04:36:46.906 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying time synchronization is in place... 2024-03-26T04:36:46.913 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T04:36:46.913 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T04:36:46.919 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T04:36:46.919 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-26T04:36:46.925 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2024-03-26T04:36:46.931 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2024-03-26T04:36:46.931 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2024-03-26T04:36:46.931 INFO:teuthology.orchestra.run.smithi038.stdout:Repeating the final host check... 2024-03-26T04:36:46.953 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:36:46.953 INFO:teuthology.orchestra.run.smithi038.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T04:36:46.953 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl is present 2024-03-26T04:36:46.953 INFO:teuthology.orchestra.run.smithi038.stdout:lvcreate is present 2024-03-26T04:36:46.959 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T04:36:46.959 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T04:36:46.965 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T04:36:46.965 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-26T04:36:46.971 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2024-03-26T04:36:46.977 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2024-03-26T04:36:46.977 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2024-03-26T04:36:46.977 INFO:teuthology.orchestra.run.smithi038.stdout:Host looks OK 2024-03-26T04:36:46.977 INFO:teuthology.orchestra.run.smithi038.stdout:Cluster fsid: 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:36:46.977 INFO:teuthology.orchestra.run.smithi038.stdout:Acquiring lock 139807903644256 on /run/cephadm/56caa506-eb2a-11ee-95d0-87774f69a715.lock 2024-03-26T04:36:46.977 INFO:teuthology.orchestra.run.smithi038.stdout:Lock 139807903644256 acquired on /run/cephadm/56caa506-eb2a-11ee-95d0-87774f69a715.lock 2024-03-26T04:36:46.978 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying IP 172.21.15.38 port 3300 ... 2024-03-26T04:36:46.978 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying IP 172.21.15.38 port 6789 ... 2024-03-26T04:36:46.978 INFO:teuthology.orchestra.run.smithi038.stdout:Base mon IP(s) is [172.21.15.38:3300, 172.21.15.38:6789], mon addrv is [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2024-03-26T04:36:46.981 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.38 metric 100 2024-03-26T04:36:46.981 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.38 metric 100 2024-03-26T04:36:46.983 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T04:36:46.984 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-26T04:36:46.984 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-26T04:36:46.986 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T04:36:46.986 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T04:36:46.986 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T04:36:46.986 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-26T04:36:46.986 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15fc/64 scope link noprefixroute 2024-03-26T04:36:46.986 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T04:36:46.987 INFO:teuthology.orchestra.run.smithi038.stdout:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2024-03-26T04:36:46.987 INFO:teuthology.orchestra.run.smithi038.stdout:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2024-03-26T04:36:46.987 INFO:teuthology.orchestra.run.smithi038.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-26T04:36:46.987 INFO:teuthology.orchestra.run.smithi038.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T04:36:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:36:47.428 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T04:36:47.428 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:36:47.428 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T04:36:47.428 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:45ddbf13772d0d6535ecfdcaa58ecfc496e5849512dfbf602c5218332a216bcc 2024-03-26T04:36:47.428 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-26T04:36:47.428 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying config sha256:5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T04:36:47.428 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T04:36:48.434 INFO:teuthology.orchestra.run.smithi038.stdout:ceph: stdout ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T04:36:48.434 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph version: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T04:36:48.434 INFO:teuthology.orchestra.run.smithi038.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T04:36:49.321 INFO:teuthology.orchestra.run.smithi038.stdout:stat: stdout 167 167 2024-03-26T04:36:49.322 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial keys... 2024-03-26T04:36:50.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQDhUAJmb+a6JxAATkARDJm/3WHk7TTt1ZXdjg== 2024-03-26T04:36:51.300 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQDiUAJmnYsDKhAAP+G4anoTP8SJA28OMB93Mw== 2024-03-26T04:36:52.353 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQDjUAJmm5tYJxAAADN119R/4EIQla8vLIlkgw== 2024-03-26T04:36:52.354 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial monmap... 2024-03-26T04:36:53.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T04:36:53.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T04:36:53.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:36:53.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T04:36:53.351 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool for a [v2:172.21.15.38:3300,v1:172.21.15.38:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T04:36:53.351 INFO:teuthology.orchestra.run.smithi038.stdout:setting min_mon_release = quincy 2024-03-26T04:36:53.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: set fsid to 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:36:53.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T04:36:53.351 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:36:53.351 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mon... 2024-03-26T04:36:54.181 INFO:teuthology.orchestra.run.smithi038.stdout:create mon.a on 2024-03-26T04:36:54.539 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T04:36:54.736 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-56caa506-eb2a-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-56caa506-eb2a-11ee-95d0-87774f69a715.target. 2024-03-26T04:36:54.736 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-56caa506-eb2a-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-56caa506-eb2a-11ee-95d0-87774f69a715.target. 2024-03-26T04:36:54.980 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.a 2024-03-26T04:36:54.980 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T04:36:55.147 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-56caa506-eb2a-11ee-95d0-87774f69a715.target.wants/ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-56caa506-eb2a-11ee-95d0-87774f69a715@.service. 2024-03-26T04:36:55.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:55 smithi038 podman[25306]: 2024-03-26 04:36:55.284667511 +0000 UTC m=+0.017375048 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:36:55.670 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:36:55.670 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-03-26T04:36:55.677 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:36:55.677 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-26T04:36:55.677 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-03-26T04:36:55.678 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T04:36:55.678 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon to start... 2024-03-26T04:36:55.678 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon... 2024-03-26T04:36:55.802 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:55 smithi038 podman[25306]: 2024-03-26 04:36:55.549497945 +0000 UTC m=+0.282205477 container init 2374a52d3b877875fa814a76932643cfddd247ba5379087a1487f313e7a6c784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:36:55.802 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:55 smithi038 podman[25306]: 2024-03-26 04:36:55.5548092 +0000 UTC m=+0.287516718 container start 2374a52d3b877875fa814a76932643cfddd247ba5379087a1487f313e7a6c784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-26T04:36:55.802 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:55 smithi038 ceph-mon[25345]: mkfs 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:36:55.802 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:55 smithi038 ceph-mon[25345]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T04:36:55.802 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:55 smithi038 bash[25306]: 2374a52d3b877875fa814a76932643cfddd247ba5379087a1487f313e7a6c784 2024-03-26T04:36:55.802 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:55 smithi038 systemd[1]: Started Ceph mon.a for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:36:56.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:56 smithi038 ceph-mon[25345]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T04:36:56.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:56 smithi038 ceph-mon[25345]: monmap e1: 1 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:36:56.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:56 smithi038 ceph-mon[25345]: fsmap 2024-03-26T04:36:56.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:56 smithi038 ceph-mon[25345]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T04:36:56.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:56 smithi038 ceph-mon[25345]: mgrmap e1: no daemons active 2024-03-26T04:36:56.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:56 smithi038 ceph-mon[25345]: from='client.? 172.21.15.38:0/2728317231' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T04:36:56.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T04:36:56.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout id: 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:36:56.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T04:36:56.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T04:36:56.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout services: 2024-03-26T04:36:56.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.611253s) 2024-03-26T04:36:56.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T04:36:56.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T04:36:56.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T04:36:56.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout data: 2024-03-26T04:36:56.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T04:36:56.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T04:36:56.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T04:36:56.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T04:36:56.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T04:36:56.836 INFO:teuthology.orchestra.run.smithi038.stdout:mon is available 2024-03-26T04:36:56.836 INFO:teuthology.orchestra.run.smithi038.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T04:36:57.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:57 smithi038 ceph-mon[25345]: from='client.? 172.21.15.38:0/4193981586' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T04:36:57.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:57 smithi038 ceph-mon[25345]: from='client.? 172.21.15.38:0/4193981586' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T04:36:57.965 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T04:36:57.965 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T04:36:57.965 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:36:57.965 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2024-03-26T04:36:57.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T04:36:57.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T04:36:57.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T04:36:57.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T04:36:57.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T04:36:57.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T04:36:57.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-03-26T04:36:57.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T04:36:57.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T04:36:57.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T04:36:57.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T04:36:57.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T04:36:57.967 INFO:teuthology.orchestra.run.smithi038.stdout:Generating new minimal ceph.conf... 2024-03-26T04:36:58.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:58 smithi038 ceph-mon[25345]: from='client.? 172.21.15.38:0/1353970764' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:58.978 INFO:teuthology.orchestra.run.smithi038.stdout:Restarting the monitor... 2024-03-26T04:36:59.263 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:58 smithi038 systemd[1]: Stopping Ceph mon.a for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:36:59.264 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:59 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a[25321]: 2024-03-26T04:36:59.221+0000 7f5fcd8c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:36:59.264 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:59 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a[25321]: 2024-03-26T04:36:59.221+0000 7f5fcd8c5700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T04:36:59.569 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:59 smithi038 podman[25679]: 2024-03-26 04:36:59.264199557 +0000 UTC m=+0.125234547 container died 2374a52d3b877875fa814a76932643cfddd247ba5379087a1487f313e7a6c784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-26T04:36:59.569 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:59 smithi038 podman[25679]: 2024-03-26 04:36:59.570144967 +0000 UTC m=+0.431179966 container cleanup 2374a52d3b877875fa814a76932643cfddd247ba5379087a1487f313e7a6c784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-26T04:36:59.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:59 smithi038 bash[25679]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a 2024-03-26T04:36:59.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:59 smithi038 podman[25698]: 2024-03-26 04:36:59.771175426 +0000 UTC m=+0.504191484 container remove 2374a52d3b877875fa814a76932643cfddd247ba5379087a1487f313e7a6c784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-26T04:37:00.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:59 smithi038 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T04:37:00.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:59 smithi038 systemd[1]: Stopped Ceph mon.a for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:37:00.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:36:59 smithi038 systemd[1]: Starting Ceph mon.a for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:37:00.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 podman[25798]: 2024-03-26 04:37:00.096826663 +0000 UTC m=+0.100765796 container create 4de8559ee353a5d7537d40bf68ffc26d4a3959a6780fe3253b9686beda204f7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:37:00.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 podman[25798]: 2024-03-26 04:37:00.013167022 +0000 UTC m=+0.017106184 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:37:00.385 INFO:teuthology.orchestra.run.smithi038.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T04:37:00.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 podman[25798]: 2024-03-26 04:37:00.290965331 +0000 UTC m=+0.294904475 container init 4de8559ee353a5d7537d40bf68ffc26d4a3959a6780fe3253b9686beda204f7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-03-26T04:37:00.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 podman[25798]: 2024-03-26 04:37:00.29656271 +0000 UTC m=+0.300501845 container start 4de8559ee353a5d7537d40bf68ffc26d4a3959a6780fe3253b9686beda204f7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124) 2024-03-26T04:37:00.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:37:00.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 7 2024-03-26T04:37:00.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: pidfile_write: ignore empty --pid-file 2024-03-26T04:37:00.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: load: jerasure load: lrc 2024-03-26T04:37:00.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:37:00.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Git sha 0 2024-03-26T04:37:00.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:37:00.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: DB SUMMARY 2024-03-26T04:37:00.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: DB Session ID: NDS6RIHBUSXAWZ3HNQYB 2024-03-26T04:37:00.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:37:00.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:37:00.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T04:37:00.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T04:37:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84629 ; 2024-03-26T04:37:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:37:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:37:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:37:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:37:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:37:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:37:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.env: 0x5586b653aae0 2024-03-26T04:37:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:37:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.info_log: 0x5586b7893920 2024-03-26T04:37:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:37:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.statistics: (nil) 2024-03-26T04:37:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.use_fsync: 0 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.db_log_dir: 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.wal_dir: 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:37:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.write_buffer_manager: 0x5586b78661e0 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:37:00.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.unordered_write: 0 2024-03-26T04:37:00.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:37:00.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:37:00.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:37:00.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:37:00.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.row_cache: None 2024-03-26T04:37:00.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.wal_filter: None 2024-03-26T04:37:00.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:37:00.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:37:00.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:37:00.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:37:00.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.wal_compression: 0 2024-03-26T04:37:00.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:37:00.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:37:00.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:37:00.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:37:00.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:37:00.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:37:00.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:37:00.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:37:00.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:37:00.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:37:00.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:37:00.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:37:00.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:37:00.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_open_files: -1 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:37:00.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Compression algorithms supported: 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: kZSTD supported: 0 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T04:37:00.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.merge_operator: 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compaction_filter: None 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5586b7893a20) 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_type: 0 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout: data_block_index_type: 0 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_shortening: 1 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: checksum: 4 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: no_block_cache: 0 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache: 0x5586b7897090 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_options: 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: capacity : 536870912 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: num_shard_bits : 4 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: strict_capacity_limit : 0 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_compressed: (nil) 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: persistent_cache: (nil) 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_size: 4096 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_size_deviation: 10 2024-03-26T04:37:00.592 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_restart_interval: 16 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_block_restart_interval: 1 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: metadata_block_size: 4096 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: partition_filters: 0 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: use_delta_encoding: 1 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: filter_policy: bloomfilter 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: whole_key_filtering: 1 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: verify_compression: 0 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: format_version: 5 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: enable_index_compression: 1 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_align: 0 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: max_auto_readahead_size: 262144 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: prepopulate_block_cache: 0 2024-03-26T04:37:00.593 INFO:journalctl@ceph.mon.a.smithi038.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:37:00.594 INFO:journalctl@ceph.mon.a.smithi038.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:37:00.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:37:00.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:37:00.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compression: NoCompression 2024-03-26T04:37:00.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:37:00.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:37:00.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:37:00.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.num_levels: 7 2024-03-26T04:37:00.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:37:00.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:37:00.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:37:00.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:37:00.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:37:00.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:37:00.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:37:00.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:37:00.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:37:00.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:37:00.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:37:00.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:37:00.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:37:00.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:37:00.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:37:00.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:37:00.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:37:00.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:37:00.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:37:00.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:37:00.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.ttl: 2592000 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:37:00.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 437412b2-5b5c-4030-a121-f75a36e65e4a 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711427820325328, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711427820326382, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81254, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79441, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73750, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711427820, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "437412b2-5b5c-4030-a121-f75a36e65e4a", "db_session_id": "NDS6RIHBUSXAWZ3HNQYB", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711427820326464, "job": 1, "event": "recovery_finished"} 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:37:00.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5586b7988000 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: DB pointer 0x5586b7972000 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** DB Stats ** 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** Compaction Stats [default] ** 2024-03-26T04:37:00.601 INFO:journalctl@ceph.mon.a.smithi038.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: L0 2/0 81.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 102.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: Sum 2/0 81.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 102.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 102.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** Compaction Stats [default] ** 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 102.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:37:00.602 INFO:journalctl@ceph.mon.a.smithi038.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative compaction: 0.00 GB write, 13.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval compaction: 0.00 GB write, 13.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout: Block cache BinnedLRUCache@0x5586b7897090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: starting mon.a rank 0 at public addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] at bind addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: mon.a@-1(???) e1 preinit fsid 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: mon.a@-1(???).mds e1 new map 2024-03-26T04:37:00.603 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: mon.a@-1(???).mds e1 print_map 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout: e1 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout: legacy client fscid: -1 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout: No filesystems configured 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 bash[25798]: 4de8559ee353a5d7537d40bf68ffc26d4a3959a6780fe3253b9686beda204f7d 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 systemd[1]: Started Ceph mon.a for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T04:37:00.604 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: monmap e1: 1 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:37:00.605 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: fsmap 2024-03-26T04:37:00.605 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T04:37:00.605 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:00 smithi038 ceph-mon[25836]: mgrmap e1: no daemons active 2024-03-26T04:37:01.466 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T04:37:01.487 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:37:01.487 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mgr... 2024-03-26T04:37:01.487 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T04:37:01.488 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T04:37:01.751 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.y 2024-03-26T04:37:01.751 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T04:37:01.903 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-56caa506-eb2a-11ee-95d0-87774f69a715.target.wants/ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-56caa506-eb2a-11ee-95d0-87774f69a715@.service. 2024-03-26T04:37:02.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:01 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2189295732' entity='client.admin' 2024-03-26T04:37:02.081 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:01 smithi038 systemd[1]: Starting Ceph mgr.y for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:37:02.450 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:02 smithi038 podman[26076]: 2024-03-26 04:37:02.139877795 +0000 UTC m=+0.114887800 container create c2e9f563228dd6f6a120cc8a70fb5e4c1c45c2b94b8769ed79b0e45bff066608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-26T04:37:02.450 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:02 smithi038 podman[26076]: 2024-03-26 04:37:02.042470604 +0000 UTC m=+0.017480635 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:37:02.450 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:02 smithi038 podman[26076]: 2024-03-26 04:37:02.342412292 +0000 UTC m=+0.317422307 container init c2e9f563228dd6f6a120cc8a70fb5e4c1c45c2b94b8769ed79b0e45bff066608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-5e70124) 2024-03-26T04:37:02.450 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:02 smithi038 podman[26076]: 2024-03-26 04:37:02.348120747 +0000 UTC m=+0.323130751 container start c2e9f563228dd6f6a120cc8a70fb5e4c1c45c2b94b8769ed79b0e45bff066608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T04:37:02.464 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:37:02.464 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-03-26T04:37:02.471 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:37:02.471 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-26T04:37:02.471 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-03-26T04:37:02.471 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T04:37:02.479 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:37:02.479 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-03-26T04:37:02.486 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:37:02.486 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-26T04:37:02.486 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-03-26T04:37:02.486 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T04:37:02.486 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr to start... 2024-03-26T04:37:02.486 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr... 2024-03-26T04:37:02.766 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:02 smithi038 bash[26076]: c2e9f563228dd6f6a120cc8a70fb5e4c1c45c2b94b8769ed79b0e45bff066608 2024-03-26T04:37:02.766 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:02 smithi038 systemd[1]: Started Ceph mgr.y for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:37:03.019 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:02 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:02.908+0000 7f2929041200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:37:03.019 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:03 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:03.005+0000 7f2929041200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:37:03.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:03 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/3463721775' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:37:03.694 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T04:37:03.694 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:37:03.694 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "56caa506-eb2a-11ee-95d0-87774f69a715", 2024-03-26T04:37:03.694 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:37:03.694 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:37:03.694 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:37:03.694 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:37:03.694 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:03.694 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:37:03.694 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:37:03.695 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:37:03.696 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:37:03.699 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:37:03.699 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:37:03.699 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:37:03.699 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:03.699 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:37:03.699 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:37:03.699 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:37:03.699 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:37:03.700 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:37:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:37:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:37:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:37:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:37:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:37:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:36:55.633799+0000", 2024-03-26T04:37:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:37:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:37:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:37:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (1/15)... 2024-03-26T04:37:04.014 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:03 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:03.678+0000 7f2929041200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:37:04.014 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:03 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:03.773+0000 7f2929041200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:37:04.014 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:03 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:03.915+0000 7f2929041200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:37:04.014 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:04 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:04.012+0000 7f2929041200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:37:04.330 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:04 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:04.097+0000 7f2929041200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:37:04.660 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:04 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:04.364+0000 7f2929041200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:37:04.660 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:04 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:04.476+0000 7f2929041200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:37:04.660 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:04 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:04.564+0000 7f2929041200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:37:04.660 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:04 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:04.660+0000 7f2929041200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:37:05.080 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:04 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:04.837+0000 7f2929041200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:37:06.075 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:05 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:05.835+0000 7f2929041200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:37:06.075 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:05 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:05.971+0000 7f2929041200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:37:06.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:06 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2295963702' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:37:06.331 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:06 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:06.320+0000 7f2929041200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:37:06.799 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:06 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:06.411+0000 7f2929041200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:37:06.799 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:06 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:06.611+0000 7f2929041200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:37:06.799 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:06 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:06.798+0000 7f2929041200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:37:06.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T04:37:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:37:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "56caa506-eb2a-11ee-95d0-87774f69a715", 2024-03-26T04:37:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:37:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:37:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:37:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:37:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:37:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:37:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:37:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:37:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:37:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:37:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:06.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:36:55.633799+0000", 2024-03-26T04:37:06.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:37:06.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:06.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:37:06.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:37:06.916 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (2/15)... 2024-03-26T04:37:07.346 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:07 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:07.086+0000 7f2929041200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:37:07.346 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:07 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:07.345+0000 7f2929041200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:37:07.768 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:07 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:07.585+0000 7f2929041200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:37:08.830 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:08 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:08.421+0000 7f2929041200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:37:09.258 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:09.002+0000 7f2929041200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:37:09.258 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:09.103+0000 7f2929041200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:37:09.581 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:09.331+0000 7f2929041200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:37:09.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-mon[25836]: Activating manager daemon y 2024-03-26T04:37:09.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-mon[25836]: mgrmap e2: y(active, starting, since 0.00760053s) 2024-03-26T04:37:09.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-mon[25836]: from='mgr.14100 172.21.15.38:0/1140095988' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:37:09.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-mon[25836]: from='mgr.14100 172.21.15.38:0/1140095988' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:37:09.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-mon[25836]: from='mgr.14100 172.21.15.38:0/1140095988' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:37:09.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-mon[25836]: from='mgr.14100 172.21.15.38:0/1140095988' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:37:09.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-mon[25836]: from='mgr.14100 172.21.15.38:0/1140095988' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:37:09.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-mon[25836]: Manager daemon y is now available 2024-03-26T04:37:09.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-mon[25836]: from='mgr.14100 172.21.15.38:0/1140095988' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:37:09.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-mon[25836]: from='mgr.14100 172.21.15.38:0/1140095988' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:37:09.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-mon[25836]: from='mgr.14100 172.21.15.38:0/1140095988' entity='mgr.y' 2024-03-26T04:37:09.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-mon[25836]: from='mgr.14100 172.21.15.38:0/1140095988' entity='mgr.y' 2024-03-26T04:37:09.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:09 smithi038 ceph-mon[25836]: from='mgr.14100 172.21.15.38:0/1140095988' entity='mgr.y' 2024-03-26T04:37:10.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T04:37:10.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:37:10.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "56caa506-eb2a-11ee-95d0-87774f69a715", 2024-03-26T04:37:10.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:37:10.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:37:10.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:37:10.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:10.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:37:10.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:10.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:37:10.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:37:10.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:37:10.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:37:10.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:37:10.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:10.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:10.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:36:55.633799+0000", 2024-03-26T04:37:10.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:37:10.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:10.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:37:10.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:37:10.045 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (3/15)... 2024-03-26T04:37:10.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:10 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/4049753684' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:37:10.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:10 smithi038 ceph-mon[25836]: mgrmap e3: y(active, since 1.01148s) 2024-03-26T04:37:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:12 smithi038 ceph-mon[25836]: mgrmap e4: y(active, since 2s) 2024-03-26T04:37:13.367 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T04:37:13.367 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:37:13.367 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "56caa506-eb2a-11ee-95d0-87774f69a715", 2024-03-26T04:37:13.367 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:37:13.367 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:37:13.367 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:37:13.367 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:37:13.367 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:13.367 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:37:13.368 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:37:13.369 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:37:13.369 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:37:13.369 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:37:13.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:13.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:37:13.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:37:13.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:37:13.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:37:13.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:37:13.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:37:13.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:37:13.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:37:13.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:37:13.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:37:13.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:37:13.374 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:13.374 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:37:13.374 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:37:13.374 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:36:55.633799+0000", 2024-03-26T04:37:13.374 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:37:13.374 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:37:13.374 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:37:13.374 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:37:13.375 INFO:teuthology.orchestra.run.smithi038.stdout:mgr is available 2024-03-26T04:37:13.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:13 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/3003720565' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:37:15.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:15 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/172048322' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T04:37:15.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:15 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/172048322' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T04:37:22.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T04:37:22.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T04:37:22.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:37:22.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T04:37:22.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T04:37:22.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T04:37:22.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T04:37:22.151 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T04:37:22.151 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T04:37:22.151 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T04:37:22.151 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T04:37:22.151 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T04:37:22.151 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T04:37:22.151 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T04:37:22.151 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling cephadm module... 2024-03-26T04:37:24.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:23 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/1386741002' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T04:37:24.331 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:24 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: ignoring --setuser ceph since I am not root 2024-03-26T04:37:24.331 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:24 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: ignoring --setgroup ceph since I am not root 2024-03-26T04:37:24.815 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:24 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:24.531+0000 7f678330a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:37:24.815 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:24 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:24.626+0000 7f678330a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:37:25.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:24 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/1386741002' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T04:37:25.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:24 smithi038 ceph-mon[25836]: mgrmap e5: y(active, since 14s) 2024-03-26T04:37:25.356 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:25 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:25.295+0000 7f678330a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:37:25.640 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:25 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:25.395+0000 7f678330a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:37:25.640 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:25 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:25.539+0000 7f678330a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:37:25.640 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:25 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:25.640+0000 7f678330a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:37:25.917 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:25 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:25.725+0000 7f678330a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:37:26.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:25 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2370879230' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T04:37:26.200 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:25 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:25.998+0000 7f678330a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:37:26.200 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:26 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:26.111+0000 7f678330a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:37:26.201 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:26 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:26.200+0000 7f678330a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:37:26.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:37:26.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T04:37:26.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:37:26.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T04:37:26.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T04:37:26.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:37:26.384 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2024-03-26T04:37:26.384 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 5... 2024-03-26T04:37:26.496 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:26 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:26.298+0000 7f678330a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:37:26.496 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:26 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:26.496+0000 7f678330a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:37:27.831 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:27 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:27.488+0000 7f678330a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:37:27.831 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:27 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:27.621+0000 7f678330a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:37:28.249 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:27 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:27.957+0000 7f678330a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:37:28.249 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:28 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:28.048+0000 7f678330a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:37:28.249 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:28 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:28.248+0000 7f678330a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:37:28.580 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:28 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:28.436+0000 7f678330a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:37:28.985 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:28 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:28.725+0000 7f678330a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:37:28.986 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:28 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:28.985+0000 7f678330a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:37:29.330 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:29 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:29.227+0000 7f678330a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:37:30.330 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:30 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:30.047+0000 7f678330a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:37:30.943 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:30 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:30.624+0000 7f678330a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:37:30.943 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:30 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:30.724+0000 7f678330a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:37:30.944 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:30 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:30.943+0000 7f678330a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:37:31.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:31 smithi038 ceph-mon[25836]: Active manager daemon y restarted 2024-03-26T04:37:31.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:31 smithi038 ceph-mon[25836]: Activating manager daemon y 2024-03-26T04:37:31.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:31 smithi038 ceph-mon[25836]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T04:37:31.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:31 smithi038 ceph-mon[25836]: mgrmap e6: y(active, starting, since 0.00799037s) 2024-03-26T04:37:31.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:31 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:37:31.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:31 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:37:31.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:31 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:37:31.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:31 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:37:31.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:31 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:37:31.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:31 smithi038 ceph-mon[25836]: Manager daemon y is now available 2024-03-26T04:37:31.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:31 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:31.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:31 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:31.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:31 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:32.286 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:32 smithi038 ceph-mon[25836]: Found migration_current of "None". Setting to last migration. 2024-03-26T04:37:32.286 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:32 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:37:32.286 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:32 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:37:32.286 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:32 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:37:32.286 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:32 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:37:32.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:32 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:37:32.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:32 smithi038 ceph-mon[25836]: mgrmap e7: y(active, since 1.00973s) 2024-03-26T04:37:32.505 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:37:32.505 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T04:37:32.505 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T04:37:32.505 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:37:32.505 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 5 is available 2024-03-26T04:37:32.505 INFO:teuthology.orchestra.run.smithi038.stdout:Setting orchestrator backend to cephadm... 2024-03-26T04:37:33.233 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:33 smithi038 ceph-mon[25836]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T04:37:33.234 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:33 smithi038 ceph-mon[25836]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T04:37:34.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:34 smithi038 ceph-mon[25836]: mgrmap e8: y(active, since 2s) 2024-03-26T04:37:34.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:34 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:34.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:34 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:34.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:34 smithi038 ceph-mon[25836]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:37:34.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:34 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:34.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:34 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:37:35.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T04:37:35.142 INFO:teuthology.orchestra.run.smithi038.stdout:Generating ssh key... 2024-03-26T04:37:35.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:35 smithi038 ceph-mon[25836]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:37:36.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-mon[25836]: [26/Mar/2024:04:37:35] ENGINE Bus STARTING 2024-03-26T04:37:36.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-mon[25836]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:37:36.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-mon[25836]: Generating ssh key... 2024-03-26T04:37:36.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-mon[25836]: [26/Mar/2024:04:37:35] ENGINE Serving on http://172.21.15.38:8765 2024-03-26T04:37:36.720 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: Generating public/private rsa key pair. 2024-03-26T04:37:36.720 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: Your identification has been saved in /tmp/tmp0fsye75r/key. 2024-03-26T04:37:36.720 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: Your public key has been saved in /tmp/tmp0fsye75r/key.pub. 2024-03-26T04:37:36.720 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: The key fingerprint is: 2024-03-26T04:37:36.720 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: SHA256:kD7YTDPZpGYFX+vz8Fu3MZWQYiayiSgRne9ZqZFZzAg ceph-56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:37:36.720 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: The key's randomart image is: 2024-03-26T04:37:36.720 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: +---[RSA 3072]----+ 2024-03-26T04:37:36.720 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: | ..Eo =.o . | 2024-03-26T04:37:36.720 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: | .o . % . . . | 2024-03-26T04:37:36.720 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: | . . #.+..+ o | 2024-03-26T04:37:36.721 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: | . .&.*+.+ . . .| 2024-03-26T04:37:36.721 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: | . .o.XoS + ..| 2024-03-26T04:37:36.721 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: | . + . = .| 2024-03-26T04:37:36.721 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: | o .o.| 2024-03-26T04:37:36.721 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: | o .+| 2024-03-26T04:37:36.721 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: | . . | 2024-03-26T04:37:36.721 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:36 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: +----[SHA256]-----+ 2024-03-26T04:37:37.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:37 smithi038 ceph-mon[25836]: [26/Mar/2024:04:37:36] ENGINE Serving on https://172.21.15.38:7150 2024-03-26T04:37:37.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:37 smithi038 ceph-mon[25836]: [26/Mar/2024:04:37:36] ENGINE Bus STARTED 2024-03-26T04:37:37.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:37 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:37:37.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:37 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:37.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:37 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:38.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:38 smithi038 ceph-mon[25836]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:37:38.211 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSpQq8lTLyE4yZa2hsGhbCrCLsTO3LjVJzwfyE9dBS0Ghm1kSQK8Y1Ydi/ifACDCQVlO8EY/IV1lCEyP6aEAIH8qNoevlZRRPhLYvtighnhX7z67uIaIWgIuoIq5wp5kpIj/o58k3O6L1sylMowFWjW0fOq1QAM1B3NwTyAEOPr0Czj0flkM1usxP+LGtlbcfg1Qusc2NJsjmM+jESZuTrxcZZHG1PTtIuT9l7A9uk/JSJxyCJz6AfaNcxJsR8lrAOascv5NMYknZ0KQe2WAC5LzXFXk5ZyYanfXh/z6cxVRkjF338AmZ4HAiXVpkMhIrBv6J/TmCGhPZ1szcP53taafDkit1Vmd2Ho6qGWqrfNnBbNf7aEEdS5QZNUOCh7C75KxvIgVopyGpvkZvNrqHhoX16fbEFPAUgWonqnZrYx1Rck5Q0SOIMMPjPHGybqqySPLV5aeUsNdmuJ2ir2APHPtjd85rp+wynZA5TOJvfB03iDk6ayYb49E1T2hTIGCU= ceph-56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:37:38.211 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:37:38.212 INFO:teuthology.orchestra.run.smithi038.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T04:37:38.212 INFO:teuthology.orchestra.run.smithi038.stdout:Adding host smithi038... 2024-03-26T04:37:39.149 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:39 smithi038 ceph-mon[25836]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:37:40.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:40 smithi038 ceph-mon[25836]: Deploying cephadm binary to smithi038 2024-03-26T04:37:43.886 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Added host 'smithi038' with addr '172.21.15.38' 2024-03-26T04:37:43.886 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying unmanaged mon service... 2024-03-26T04:37:44.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:44 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:44.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:44 smithi038 ceph-mon[25836]: Added host smithi038 2024-03-26T04:37:44.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:44 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:37:45.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T04:37:45.438 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying unmanaged mgr service... 2024-03-26T04:37:45.762 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:45 smithi038 ceph-mon[25836]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:37:45.762 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:45 smithi038 ceph-mon[25836]: Saving service mon spec with placement count:5 2024-03-26T04:37:45.762 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:45 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:46.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T04:37:47.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:47 smithi038 ceph-mon[25836]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:37:47.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:47 smithi038 ceph-mon[25836]: Saving service mgr spec with placement count:2 2024-03-26T04:37:47.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:47 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:47.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:47 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:48.667 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:48 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:48.667 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:48 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/217522668' entity='client.admin' 2024-03-26T04:37:49.693 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling the dashboard module... 2024-03-26T04:37:50.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:50 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/719561909' entity='client.admin' 2024-03-26T04:37:51.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:51 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:51.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:51 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:51.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:51 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:37:51.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:51 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' 2024-03-26T04:37:51.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:51 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi038", "caps": []}]: dispatch 2024-03-26T04:37:51.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:51 smithi038 ceph-mon[25836]: from='mgr.14120 172.21.15.38:0/1425524340' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi038", "caps": []}]': finished 2024-03-26T04:37:51.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:51 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2325887958' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T04:37:51.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:51 smithi038 ceph-mon[25836]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:37:51.831 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:51 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: ignoring --setuser ceph since I am not root 2024-03-26T04:37:51.831 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:51 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: ignoring --setgroup ceph since I am not root 2024-03-26T04:37:52.331 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:52 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:52.173+0000 7efecc33e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:37:52.331 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:52 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:52.269+0000 7efecc33e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:37:52.628 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:52 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2325887958' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T04:37:52.628 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:52 smithi038 ceph-mon[25836]: mgrmap e9: y(active, since 20s) 2024-03-26T04:37:53.292 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:52 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:52.949+0000 7efecc33e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:37:53.292 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:53 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:53.045+0000 7efecc33e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:37:53.292 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:53 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:53.192+0000 7efecc33e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:37:53.293 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:53 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:53.291+0000 7efecc33e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:37:53.580 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:53 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:53.378+0000 7efecc33e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:37:53.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:37:53.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T04:37:53.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:37:53.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T04:37:53.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T04:37:53.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:37:53.760 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2024-03-26T04:37:53.760 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 9... 2024-03-26T04:37:53.855 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:53 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2253655786' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T04:37:53.855 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:53 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:53.648+0000 7efecc33e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:37:53.855 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:53 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:53.762+0000 7efecc33e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:37:53.855 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:53 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:53.855+0000 7efecc33e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:37:54.125 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:53 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:53.953+0000 7efecc33e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:37:54.580 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:54 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:54.137+0000 7efecc33e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:37:55.580 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:55 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:55.127+0000 7efecc33e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:37:55.581 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:55 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:55.262+0000 7efecc33e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:37:55.901 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:55 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:55.606+0000 7efecc33e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:37:55.901 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:55 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:55.698+0000 7efecc33e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:37:55.901 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:55 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:55.901+0000 7efecc33e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:37:56.330 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:56 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:56.093+0000 7efecc33e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:37:56.650 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:56 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:56.388+0000 7efecc33e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:37:56.650 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:56 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:56.650+0000 7efecc33e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:37:57.081 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:56 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:56.898+0000 7efecc33e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:37:58.080 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:57 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:57.730+0000 7efecc33e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:37:58.580 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:58.310+0000 7efecc33e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:37:58.581 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:58.411+0000 7efecc33e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:37:59.080 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:37:58.633+0000 7efecc33e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:37:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-mon[25836]: Active manager daemon y restarted 2024-03-26T04:37:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-mon[25836]: Activating manager daemon y 2024-03-26T04:37:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-mon[25836]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T04:37:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-mon[25836]: mgrmap e10: y(active, starting, since 0.00818782s) 2024-03-26T04:37:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:37:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:37:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:37:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:37:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:37:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-mon[25836]: Manager daemon y is now available 2024-03-26T04:37:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:58 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:37:59.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:59 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:37:59.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:59 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:37:59.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:59 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:37:59.964 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:59 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:37:59.964 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:59 smithi038 ceph-mon[25836]: [26/Mar/2024:04:37:59] ENGINE Bus STARTING 2024-03-26T04:37:59.964 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:59 smithi038 ceph-mon[25836]: [26/Mar/2024:04:37:59] ENGINE Serving on http://172.21.15.38:8765 2024-03-26T04:37:59.964 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:59 smithi038 ceph-mon[25836]: [26/Mar/2024:04:37:59] ENGINE Serving on https://172.21.15.38:7150 2024-03-26T04:37:59.964 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:59 smithi038 ceph-mon[25836]: [26/Mar/2024:04:37:59] ENGINE Bus STARTED 2024-03-26T04:37:59.964 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:37:59 smithi038 ceph-mon[25836]: mgrmap e11: y(active, since 1.01046s) 2024-03-26T04:38:00.216 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:38:00.216 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T04:38:00.216 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T04:38:00.216 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:38:00.216 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 9 is available 2024-03-26T04:38:00.216 INFO:teuthology.orchestra.run.smithi038.stdout:Generating a dashboard self-signed certificate... 2024-03-26T04:38:01.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:01 smithi038 ceph-mon[25836]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T04:38:01.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:01 smithi038 ceph-mon[25836]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T04:38:01.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:01.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:38:01.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:01.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi038", "caps": []}]: dispatch 2024-03-26T04:38:01.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:01.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:01.973 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T04:38:01.973 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial admin user... 2024-03-26T04:38:02.907 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:02 smithi038 ceph-mon[25836]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:38:02.907 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:02 smithi038 ceph-mon[25836]: mgrmap e12: y(active, since 3s) 2024-03-26T04:38:03.446 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$IRRNWiOLS0/4czEJljYD0O6fKiez3D2NRAkytD5qiu3F/Id7JawlO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711427882, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T04:38:03.446 INFO:teuthology.orchestra.run.smithi038.stdout:Fetching dashboard port number... 2024-03-26T04:38:03.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:03 smithi038 ceph-mon[25836]: Deploying daemon agent.smithi038 on smithi038 2024-03-26T04:38:03.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:03 smithi038 ceph-mon[25836]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:38:03.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:03 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:04.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:04 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/495698021' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T04:38:04.736 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T04:38:04.745 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:38:04.745 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-03-26T04:38:04.751 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:38:04.751 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-26T04:38:04.751 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-03-26T04:38:04.751 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T04:38:04.753 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph Dashboard is now available at: 2024-03-26T04:38:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:38:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: URL: https://smithi038.front.sepia.ceph.com:8443/ 2024-03-26T04:38:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: User: admin 2024-03-26T04:38:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: Password: 866quddqfl 2024-03-26T04:38:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:38:04.754 INFO:teuthology.orchestra.run.smithi038.stdout:Saving cluster configuration to /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config directory 2024-03-26T04:38:04.754 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling autotune for osd_memory_target 2024-03-26T04:38:06.770 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:06 smithi038 ceph-mon[25836]: mgrmap e13: y(active, since 7s) 2024-03-26T04:38:07.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T04:38:07.314 INFO:teuthology.orchestra.run.smithi038.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout: ceph telemetry on 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout:For more information see: 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T04:38:07.315 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:38:07.316 INFO:teuthology.orchestra.run.smithi038.stdout:Bootstrap complete. 2024-03-26T04:38:07.360 INFO:tasks.cephadm:Fetching config... 2024-03-26T04:38:07.361 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:38:07.361 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T04:38:07.376 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T04:38:07.376 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:38:07.376 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T04:38:07.432 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T04:38:07.432 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:38:07.432 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T04:38:07.587 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T04:38:07.587 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:38:07.587 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T04:38:07.644 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T04:38:07.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSpQq8lTLyE4yZa2hsGhbCrCLsTO3LjVJzwfyE9dBS0Ghm1kSQK8Y1Ydi/ifACDCQVlO8EY/IV1lCEyP6aEAIH8qNoevlZRRPhLYvtighnhX7z67uIaIWgIuoIq5wp5kpIj/o58k3O6L1sylMowFWjW0fOq1QAM1B3NwTyAEOPr0Czj0flkM1usxP+LGtlbcfg1Qusc2NJsjmM+jESZuTrxcZZHG1PTtIuT9l7A9uk/JSJxyCJz6AfaNcxJsR8lrAOascv5NMYknZ0KQe2WAC5LzXFXk5ZyYanfXh/z6cxVRkjF338AmZ4HAiXVpkMhIrBv6J/TmCGhPZ1szcP53taafDkit1Vmd2Ho6qGWqrfNnBbNf7aEEdS5QZNUOCh7C75KxvIgVopyGpvkZvNrqHhoX16fbEFPAUgWonqnZrYx1Rck5Q0SOIMMPjPHGybqqySPLV5aeUsNdmuJ2ir2APHPtjd85rp+wynZA5TOJvfB03iDk6ayYb49E1T2hTIGCU= ceph-56caa506-eb2a-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:38:07.793 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSpQq8lTLyE4yZa2hsGhbCrCLsTO3LjVJzwfyE9dBS0Ghm1kSQK8Y1Ydi/ifACDCQVlO8EY/IV1lCEyP6aEAIH8qNoevlZRRPhLYvtighnhX7z67uIaIWgIuoIq5wp5kpIj/o58k3O6L1sylMowFWjW0fOq1QAM1B3NwTyAEOPr0Czj0flkM1usxP+LGtlbcfg1Qusc2NJsjmM+jESZuTrxcZZHG1PTtIuT9l7A9uk/JSJxyCJz6AfaNcxJsR8lrAOascv5NMYknZ0KQe2WAC5LzXFXk5ZyYanfXh/z6cxVRkjF338AmZ4HAiXVpkMhIrBv6J/TmCGhPZ1szcP53taafDkit1Vmd2Ho6qGWqrfNnBbNf7aEEdS5QZNUOCh7C75KxvIgVopyGpvkZvNrqHhoX16fbEFPAUgWonqnZrYx1Rck5Q0SOIMMPjPHGybqqySPLV5aeUsNdmuJ2ir2APHPtjd85rp+wynZA5TOJvfB03iDk6ayYb49E1T2hTIGCU= ceph-56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:38:07.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSpQq8lTLyE4yZa2hsGhbCrCLsTO3LjVJzwfyE9dBS0Ghm1kSQK8Y1Ydi/ifACDCQVlO8EY/IV1lCEyP6aEAIH8qNoevlZRRPhLYvtighnhX7z67uIaIWgIuoIq5wp5kpIj/o58k3O6L1sylMowFWjW0fOq1QAM1B3NwTyAEOPr0Czj0flkM1usxP+LGtlbcfg1Qusc2NJsjmM+jESZuTrxcZZHG1PTtIuT9l7A9uk/JSJxyCJz6AfaNcxJsR8lrAOascv5NMYknZ0KQe2WAC5LzXFXk5ZyYanfXh/z6cxVRkjF338AmZ4HAiXVpkMhIrBv6J/TmCGhPZ1szcP53taafDkit1Vmd2Ho6qGWqrfNnBbNf7aEEdS5QZNUOCh7C75KxvIgVopyGpvkZvNrqHhoX16fbEFPAUgWonqnZrYx1Rck5Q0SOIMMPjPHGybqqySPLV5aeUsNdmuJ2ir2APHPtjd85rp+wynZA5TOJvfB03iDk6ayYb49E1T2hTIGCU= ceph-56caa506-eb2a-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:38:07.853 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSpQq8lTLyE4yZa2hsGhbCrCLsTO3LjVJzwfyE9dBS0Ghm1kSQK8Y1Ydi/ifACDCQVlO8EY/IV1lCEyP6aEAIH8qNoevlZRRPhLYvtighnhX7z67uIaIWgIuoIq5wp5kpIj/o58k3O6L1sylMowFWjW0fOq1QAM1B3NwTyAEOPr0Czj0flkM1usxP+LGtlbcfg1Qusc2NJsjmM+jESZuTrxcZZHG1PTtIuT9l7A9uk/JSJxyCJz6AfaNcxJsR8lrAOascv5NMYknZ0KQe2WAC5LzXFXk5ZyYanfXh/z6cxVRkjF338AmZ4HAiXVpkMhIrBv6J/TmCGhPZ1szcP53taafDkit1Vmd2Ho6qGWqrfNnBbNf7aEEdS5QZNUOCh7C75KxvIgVopyGpvkZvNrqHhoX16fbEFPAUgWonqnZrYx1Rck5Q0SOIMMPjPHGybqqySPLV5aeUsNdmuJ2ir2APHPtjd85rp+wynZA5TOJvfB03iDk6ayYb49E1T2hTIGCU= ceph-56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:38:07.868 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSpQq8lTLyE4yZa2hsGhbCrCLsTO3LjVJzwfyE9dBS0Ghm1kSQK8Y1Ydi/ifACDCQVlO8EY/IV1lCEyP6aEAIH8qNoevlZRRPhLYvtighnhX7z67uIaIWgIuoIq5wp5kpIj/o58k3O6L1sylMowFWjW0fOq1QAM1B3NwTyAEOPr0Czj0flkM1usxP+LGtlbcfg1Qusc2NJsjmM+jESZuTrxcZZHG1PTtIuT9l7A9uk/JSJxyCJz6AfaNcxJsR8lrAOascv5NMYknZ0KQe2WAC5LzXFXk5ZyYanfXh/z6cxVRkjF338AmZ4HAiXVpkMhIrBv6J/TmCGhPZ1szcP53taafDkit1Vmd2Ho6qGWqrfNnBbNf7aEEdS5QZNUOCh7C75KxvIgVopyGpvkZvNrqHhoX16fbEFPAUgWonqnZrYx1Rck5Q0SOIMMPjPHGybqqySPLV5aeUsNdmuJ2ir2APHPtjd85rp+wynZA5TOJvfB03iDk6ayYb49E1T2hTIGCU= ceph-56caa506-eb2a-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:38:07.893 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:07 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/1679474134' entity='client.admin' 2024-03-26T04:38:07.914 INFO:teuthology.orchestra.run.smithi177.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSpQq8lTLyE4yZa2hsGhbCrCLsTO3LjVJzwfyE9dBS0Ghm1kSQK8Y1Ydi/ifACDCQVlO8EY/IV1lCEyP6aEAIH8qNoevlZRRPhLYvtighnhX7z67uIaIWgIuoIq5wp5kpIj/o58k3O6L1sylMowFWjW0fOq1QAM1B3NwTyAEOPr0Czj0flkM1usxP+LGtlbcfg1Qusc2NJsjmM+jESZuTrxcZZHG1PTtIuT9l7A9uk/JSJxyCJz6AfaNcxJsR8lrAOascv5NMYknZ0KQe2WAC5LzXFXk5ZyYanfXh/z6cxVRkjF338AmZ4HAiXVpkMhIrBv6J/TmCGhPZ1szcP53taafDkit1Vmd2Ho6qGWqrfNnBbNf7aEEdS5QZNUOCh7C75KxvIgVopyGpvkZvNrqHhoX16fbEFPAUgWonqnZrYx1Rck5Q0SOIMMPjPHGybqqySPLV5aeUsNdmuJ2ir2APHPtjd85rp+wynZA5TOJvfB03iDk6ayYb49E1T2hTIGCU= ceph-56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:38:07.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T04:38:08.282 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:38:09.394 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:09 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:09.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:09 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:09.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:09 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:09.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:09 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:09.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:09 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:09.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:09 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:09.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:09 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:09.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:09 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:09.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:09 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:09.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:09 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:09.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:09 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:09.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:09 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:09.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:09 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:10.822 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T04:38:10.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T04:38:11.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:11 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/3523098284' entity='client.admin' 2024-03-26T04:38:11.507 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:11 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:11.507 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:11 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:12.718 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:38:13.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:13 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:13.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:13 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:15.563 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:15 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:15.563 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:15 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:15.884 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi114 2024-03-26T04:38:15.885 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-26T04:38:15.885 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.conf 2024-03-26T04:38:15.904 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-26T04:38:15.904 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:38:15.961 INFO:tasks.cephadm:Adding host smithi114 to orchestrator... 2024-03-26T04:38:15.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch host add smithi114 2024-03-26T04:38:16.206 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:38:16.448 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:16 smithi038 ceph-mon[25836]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:38:16.448 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:16 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:16.448 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:16 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:16.448 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:16 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:16.448 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:16 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:16.448 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:16 smithi038 ceph-mon[25836]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T04:38:16.449 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:16 smithi038 ceph-mon[25836]: Updating smithi038:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:38:16.449 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:16 smithi038 ceph-mon[25836]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:38:16.449 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:16 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:16.449 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:16 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:16.449 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:16 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:17.320 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:17 smithi038 ceph-mon[25836]: Updating smithi038:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:38:19.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:18 smithi038 ceph-mon[25836]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:38:20.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:19 smithi038 ceph-mon[25836]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:20.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:19 smithi038 ceph-mon[25836]: Deploying cephadm binary to smithi114 2024-03-26T04:38:22.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:21 smithi038 ceph-mon[25836]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:22.443 INFO:teuthology.orchestra.run.smithi038.stdout:Added host 'smithi114' with addr '172.21.15.114' 2024-03-26T04:38:23.049 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T04:38:23.290 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:38:23.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:23 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:23.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:23 smithi038 ceph-mon[25836]: Added host smithi114 2024-03-26T04:38:23.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:23 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:23.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:23 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:24 smithi038 ceph-mon[25836]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:38:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:24.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:24.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:24 smithi038 ceph-mon[25836]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-26T04:38:24.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:24.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:24.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi114", "caps": []}]: dispatch 2024-03-26T04:38:24.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi114", "caps": []}]': finished 2024-03-26T04:38:25.201 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:38:25.201 INFO:teuthology.orchestra.run.smithi038.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}] 2024-03-26T04:38:25.783 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:25 smithi038 ceph-mon[25836]: Updating smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:38:25.783 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:25 smithi038 ceph-mon[25836]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:38:25.783 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:25 smithi038 ceph-mon[25836]: Updating smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:38:25.783 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:25 smithi038 ceph-mon[25836]: Deploying daemon agent.smithi114 on smithi114 2024-03-26T04:38:25.808 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi177 2024-03-26T04:38:25.808 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-26T04:38:25.808 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.conf 2024-03-26T04:38:25.827 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-26T04:38:25.828 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:38:25.885 INFO:tasks.cephadm:Adding host smithi177 to orchestrator... 2024-03-26T04:38:25.885 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch host add smithi177 2024-03-26T04:38:26.122 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:38:26.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:26.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:38:26.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:26.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:26.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:26.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:26.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:26.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:26.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:26.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:26.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:26.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:26.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:26.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:26.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:26.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:26 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:28.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:28 smithi038 ceph-mon[25836]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:29.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:29 smithi038 ceph-mon[25836]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:38:29.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:29 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:29.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:29 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:29.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:29 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:30 smithi038 ceph-mon[25836]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:30 smithi038 ceph-mon[25836]: Deploying cephadm binary to smithi177 2024-03-26T04:38:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:30 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:30 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:30 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:32.488 INFO:teuthology.orchestra.run.smithi038.stdout:Added host 'smithi177' with addr '172.21.15.177' 2024-03-26T04:38:32.646 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:32 smithi038 ceph-mon[25836]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:32.646 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:32 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:32.646 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:32 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:33.228 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T04:38:33.468 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:38:33.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:33 smithi038 ceph-mon[25836]: Added host smithi177 2024-03-26T04:38:33.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:33 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:33.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:33 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:33.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:33 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:33.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:33 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:38:33.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:33 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:33.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:33 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:34.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:34 smithi038 ceph-mon[25836]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:34.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:34 smithi038 ceph-mon[25836]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-26T04:38:34.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:34 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:34.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:34 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:34.755 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:34 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]: dispatch 2024-03-26T04:38:34.755 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:34 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]': finished 2024-03-26T04:38:35.324 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:38:35.324 INFO:teuthology.orchestra.run.smithi038.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.177", "hostname": "smithi177", "labels": [], "status": ""}] 2024-03-26T04:38:35.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:35 smithi038 ceph-mon[25836]: Updating smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:38:35.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:35 smithi038 ceph-mon[25836]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:38:35.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:35 smithi038 ceph-mon[25836]: Updating smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:38:35.940 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T04:38:35.941 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T04:38:36.183 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:38:36.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:36.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: Deploying daemon agent.smithi177 on smithi177 2024-03-26T04:38:36.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:38:36.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:36.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:36.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:36.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:36.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:36.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:36.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:36.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:36.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:36.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:36.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:36.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:37.757 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:37 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:37.757 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:37 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:38.628 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2024-03-26T04:38:38.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:38 smithi038 ceph-mon[25836]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:38.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:38 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/449518722' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T04:38:39.220 INFO:tasks.cephadm:Adding mon.a on smithi038 2024-03-26T04:38:39.221 INFO:tasks.cephadm:Adding mon.b on smithi114 2024-03-26T04:38:39.221 INFO:tasks.cephadm:Adding mon.c on smithi177 2024-03-26T04:38:39.221 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi038:172.21.15.38=a;smithi114:172.21.15.114=b;smithi177:172.21.15.177=c' 2024-03-26T04:38:39.465 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:38:40.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:39 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/449518722' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T04:38:40.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:39 smithi038 ceph-mon[25836]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:38:40.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:39 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:40.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:39 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:40.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:39 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:41.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:40 smithi038 ceph-mon[25836]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:41.125 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mon update... 2024-03-26T04:38:41.665 DEBUG:teuthology.orchestra.run.smithi114:mon.b> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T04:38:41.668 DEBUG:teuthology.orchestra.run.smithi177:mon.c> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T04:38:41.670 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:38:41.670 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:38:42.403 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:38:42.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:42 smithi038 ceph-mon[25836]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:42.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:42 smithi038 ceph-mon[25836]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:172.21.15.38=a;smithi114:172.21.15.114=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:38:42.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:42 smithi038 ceph-mon[25836]: Saving service mon spec with placement smithi038:172.21.15.38=a;smithi114:172.21.15.114=b;smithi177:172.21.15.177=c;count:3 2024-03-26T04:38:42.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:42 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:42.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:42 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:42.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:42 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:42.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:42 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:42.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:42 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:42.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:42 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:38:42.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:42 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:42.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:42 smithi038 ceph-mon[25836]: Deploying daemon mon.c on smithi177 2024-03-26T04:38:44.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:44 smithi038 ceph-mon[25836]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:45.577 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:38:45.577 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"56caa506-eb2a-11ee-95d0-87774f69a715","modified":"2024-03-26T04:36:52.730001Z","created":"2024-03-26T04:36:52.730001Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T04:38:45.579 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-26T04:38:47.426 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:38:47.427 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:38:47.662 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:38:51.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:38:51.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:51.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: mon.a calling monitor election 2024-03-26T04:38:51.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:51.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:51.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:51.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: mon.c calling monitor election 2024-03-26T04:38:51.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:51.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:51.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:51.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:51.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:38:51.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: monmap e2: 2 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:38:51.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: fsmap 2024-03-26T04:38:51.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:38:51.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: mgrmap e13: y(active, since 52s) 2024-03-26T04:38:51.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: overall HEALTH_OK 2024-03-26T04:38:51.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:51.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:51.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:51.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:51.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:51.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:51.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:51.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:51.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:38:50 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:52.281 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:51 smithi038 ceph-mon[25836]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:52.281 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:51 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:52.281 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:51 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:38:52.281 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:51 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:52.281 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:51 smithi038 ceph-mon[25836]: Deploying daemon mon.b on smithi114 2024-03-26T04:38:52.281 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:51 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:52.347 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:38:52.348 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":2,"fsid":"56caa506-eb2a-11ee-95d0-87774f69a715","modified":"2024-03-26T04:38:45.890531Z","created":"2024-03-26T04:36:52.730001Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T04:38:52.348 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:38:51 smithi177 ceph-mon[27677]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:52.348 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:38:51 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:52.348 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:38:51 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:38:52.348 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:38:51 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:52.348 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:38:51 smithi177 ceph-mon[27677]: Deploying daemon mon.b on smithi114 2024-03-26T04:38:52.348 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:38:51 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:52.350 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 2 2024-03-26T04:38:53.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:53 smithi038 ceph-mon[25836]: from='client.? 172.21.15.177:0/2427752007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:38:53.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:38:53 smithi177 ceph-mon[27677]: from='client.? 172.21.15.177:0/2427752007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:38:53.923 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:38:53.923 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:38:54.168 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:38:54.286 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:38:54 smithi177 ceph-mon[27677]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:54.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:38:54 smithi038 ceph-mon[25836]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:54.776 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 podman[27589]: 2024-03-26 04:38:54.656581418 +0000 UTC m=+0.118082898 container create 11c8da4b99a3c6a4dfe5c834f881bbb8b331d324f41045be8221618dd2f298b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-b, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:38:55.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 podman[27589]: 2024-03-26 04:38:54.867462787 +0000 UTC m=+0.328964280 container init 11c8da4b99a3c6a4dfe5c834f881bbb8b331d324f41045be8221618dd2f298b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, ceph=True, org.label-schema.build-date=20240319) 2024-03-26T04:38:55.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 podman[27589]: 2024-03-26 04:38:54.872371213 +0000 UTC m=+0.333872694 container start 11c8da4b99a3c6a4dfe5c834f881bbb8b331d324f41045be8221618dd2f298b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-b, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T04:38:55.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:38:55.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 7 2024-03-26T04:38:55.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pidfile_write: ignore empty --pid-file 2024-03-26T04:38:55.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: load: jerasure load: lrc 2024-03-26T04:38:55.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:38:55.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Git sha 0 2024-03-26T04:38:55.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:38:55.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: DB SUMMARY 2024-03-26T04:38:55.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: DB Session ID: Z0FNGYETF1SDR5FSIM04 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.env: 0x55755ea8aae0 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:38:55.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.info_log: 0x5575601a9920 2024-03-26T04:38:55.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:38:55.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.statistics: (nil) 2024-03-26T04:38:55.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.use_fsync: 0 2024-03-26T04:38:55.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:38:55.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:38:55.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:38:55.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:38:55.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:38:55.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:38:55.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:38:55.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:38:55.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:38:55.150 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:38:55.150 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:38:55.150 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.db_log_dir: 2024-03-26T04:38:55.150 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.wal_dir: 2024-03-26T04:38:55.150 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:38:55.150 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:38:55.150 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:38:55.150 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:38:55.150 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:38:55.150 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:38:55.150 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:38:55.150 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:38:55.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.write_buffer_manager: 0x55756017c1e0 2024-03-26T04:38:55.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:38:55.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:38:55.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:38:55.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:38:55.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:38:55.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:38:55.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:38:55.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:38:55.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.unordered_write: 0 2024-03-26T04:38:55.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:38:55.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:38:55.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:38:55.152 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:38:55.152 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.row_cache: None 2024-03-26T04:38:55.152 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.wal_filter: None 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.wal_compression: 0 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:38:55.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:38:55.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_open_files: -1 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Compression algorithms supported: 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: kZSTD supported: 0 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:38:55.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.merge_operator: 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compaction_filter: None 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5575601a9a20) 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:38:55.157 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_type: 0 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: data_block_index_type: 0 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_shortening: 1 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: checksum: 4 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: no_block_cache: 0 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache: 0x5575601ad090 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_options: 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: capacity : 536870912 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: num_shard_bits : 4 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: strict_capacity_limit : 0 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_compressed: (nil) 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: persistent_cache: (nil) 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_size: 4096 2024-03-26T04:38:55.158 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_size_deviation: 10 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_restart_interval: 16 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_block_restart_interval: 1 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: metadata_block_size: 4096 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: partition_filters: 0 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: use_delta_encoding: 1 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: filter_policy: bloomfilter 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: whole_key_filtering: 1 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: verify_compression: 0 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: format_version: 5 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: enable_index_compression: 1 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_align: 0 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: max_auto_readahead_size: 262144 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: prepopulate_block_cache: 0 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:38:55.159 INFO:journalctl@ceph.mon.b.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compression: NoCompression 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.num_levels: 7 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:38:55.160 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:38:55.161 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.ttl: 2592000 2024-03-26T04:38:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T04:38:55.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 10e11237-1eed-4f3a-a6d9-20beef20fcfa 2024-03-26T04:38:55.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711427934901728, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T04:38:55.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T04:38:55.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711427934902139, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711427934, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "10e11237-1eed-4f3a-a6d9-20beef20fcfa", "db_session_id": "Z0FNGYETF1SDR5FSIM04", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:38:55.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711427934902218, "job": 1, "event": "recovery_finished"} 2024-03-26T04:38:55.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T04:38:55.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:38:55.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55756029e000 2024-03-26T04:38:55.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: DB pointer 0x557560288000 2024-03-26T04:38:55.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:38:55.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:38:55.166 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** DB Stats ** 2024-03-26T04:38:55.166 INFO:journalctl@ceph.mon.b.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:38:55.166 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2024-03-26T04:38:55.167 INFO:journalctl@ceph.mon.b.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:38:55.168 INFO:journalctl@ceph.mon.b.smithi114.stdout: Block cache BinnedLRUCache@0x5575601ad090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: using public_addr v2:172.21.15.114:0/0 -> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: starting mon.b rank -1 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(???) e0 preinit fsid 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(synchronizing).mds e1 new map 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout: e1 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout: legacy client fscid: -1 2024-03-26T04:38:55.169 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout: No filesystems configured 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Added host smithi114 2024-03-26T04:38:55.170 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi114", "caps": []}]: dispatch 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi114", "caps": []}]': finished 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Updating smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:38:55.171 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Updating smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Deploying daemon agent.smithi114 on smithi114 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.172 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.173 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.173 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.173 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:38:55.173 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.173 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.173 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.173 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.173 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Deploying cephadm binary to smithi177 2024-03-26T04:38:55.173 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.173 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.173 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.173 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.173 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Added host smithi177 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]: dispatch 2024-03-26T04:38:55.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]': finished 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Updating smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Updating smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Deploying daemon agent.smithi177 on smithi177 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:55.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/449518722' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/449518722' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.176 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:172.21.15.38=a;smithi114:172.21.15.114=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Saving service mon spec with placement smithi038:172.21.15.38=a;smithi114:172.21.15.114=b;smithi177:172.21.15.177=c;count:3 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Deploying daemon mon.c on smithi177 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:55.177 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.a calling monitor election 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.c calling monitor election 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: monmap e2: 2 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: fsmap 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mgrmap e13: y(active, since 52s) 2024-03-26T04:38:55.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: overall HEALTH_OK 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: Deploying daemon mon.b on smithi114 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: from='client.? 172.21.15.177:0/2427752007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:38:55.179 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:38:55.180 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 ceph-mon[27627]: mon.b@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-26T04:38:55.180 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 bash[27589]: 11c8da4b99a3c6a4dfe5c834f881bbb8b331d324f41045be8221618dd2f298b6 2024-03-26T04:38:55.180 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:38:54 smithi114 systemd[1]: Started Ceph mon.b for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:39:00.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:00.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:39:00.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:00.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: mon.a calling monitor election 2024-03-26T04:39:00.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: mon.c calling monitor election 2024-03-26T04:39:00.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: fsmap 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: mgrmap e13: y(active, since 61s) 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:39:00.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:39:00.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: mon.b (rank 2) addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] is down (out of quorum) 2024-03-26T04:39:00.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:00.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:00.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:00.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:00.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:00 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:00.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:00.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:39:00.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:00.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: mon.a calling monitor election 2024-03-26T04:39:00.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: mon.c calling monitor election 2024-03-26T04:39:00.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:39:00.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:00.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:00.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:00.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:00.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: fsmap 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: mgrmap e13: y(active, since 61s) 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: mon.b (rank 2) addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] is down (out of quorum) 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:00.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:00.369 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:00.369 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:00 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-26T04:39:00.369 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":3,"fsid":"56caa506-eb2a-11ee-95d0-87774f69a715","modified":"2024-03-26T04:38:54.928133Z","created":"2024-03-26T04:36:52.730001Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T04:39:00.369 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 3 2024-03-26T04:39:00.988 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T04:39:00.988 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T04:39:01.240 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:39:01.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:01.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:01.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:01.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T04:39:01.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-26T04:39:01.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='client.? 172.21.15.177:0/306739798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: Updating smithi038:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: Updating smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: Updating smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:39:01.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:01.255 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:01 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:01.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:01.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:01.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:01.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T04:39:01.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-26T04:39:01.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-26T04:39:01.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='client.? 172.21.15.177:0/306739798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:39:01.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: Updating smithi038:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:39:01.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: Updating smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:39:01.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: Updating smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:39:01.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:01.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:39:01.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:39:01.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:01.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:01 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:02.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:02.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:39:02.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:02.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: mon.a calling monitor election 2024-03-26T04:39:02.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: mon.c calling monitor election 2024-03-26T04:39:02.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:39:02.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:02.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:02.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:02.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:02.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:02.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:02.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:02.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:39:02.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:39:02.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: fsmap 2024-03-26T04:39:02.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:39:02.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: mgrmap e13: y(active, since 61s) 2024-03-26T04:39:02.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:39:02.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: mon.b (rank 2) addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] is down (out of quorum) 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-26T04:39:02.148 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='client.? 172.21.15.177:0/306739798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: Updating smithi038:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: Updating smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: Updating smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:02.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:01 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:02.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:02 smithi177 ceph-mon[27677]: mon.b calling monitor election 2024-03-26T04:39:02.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:02 smithi177 ceph-mon[27677]: mon.b calling monitor election 2024-03-26T04:39:02.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:02 smithi177 ceph-mon[27677]: mon.a calling monitor election 2024-03-26T04:39:02.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:02 smithi177 ceph-mon[27677]: mon.c calling monitor election 2024-03-26T04:39:02.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:02 smithi177 ceph-mon[27677]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:39:02.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:02 smithi177 ceph-mon[27677]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:39:02.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:02 smithi177 ceph-mon[27677]: fsmap 2024-03-26T04:39:02.254 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:02 smithi177 ceph-mon[27677]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:39:02.255 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:02 smithi177 ceph-mon[27677]: mgrmap e13: y(active, since 63s) 2024-03-26T04:39:02.255 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:02 smithi177 ceph-mon[27677]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T04:39:02.255 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:02 smithi177 ceph-mon[27677]: Cluster is now healthy 2024-03-26T04:39:02.255 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:02 smithi177 ceph-mon[27677]: overall HEALTH_OK 2024-03-26T04:39:02.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:02 smithi038 ceph-mon[25836]: mon.b calling monitor election 2024-03-26T04:39:02.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:02 smithi038 ceph-mon[25836]: mon.b calling monitor election 2024-03-26T04:39:02.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:02 smithi038 ceph-mon[25836]: mon.a calling monitor election 2024-03-26T04:39:02.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:02 smithi038 ceph-mon[25836]: mon.c calling monitor election 2024-03-26T04:39:02.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:02 smithi038 ceph-mon[25836]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:39:02.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:02 smithi038 ceph-mon[25836]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:39:02.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:02 smithi038 ceph-mon[25836]: fsmap 2024-03-26T04:39:02.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:02 smithi038 ceph-mon[25836]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:39:02.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:02 smithi038 ceph-mon[25836]: mgrmap e13: y(active, since 63s) 2024-03-26T04:39:02.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:02 smithi038 ceph-mon[25836]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T04:39:02.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:02 smithi038 ceph-mon[25836]: Cluster is now healthy 2024-03-26T04:39:02.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:02 smithi038 ceph-mon[25836]: overall HEALTH_OK 2024-03-26T04:39:02.481 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:02 smithi114 ceph-mon[27627]: mon.b calling monitor election 2024-03-26T04:39:02.482 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:02 smithi114 ceph-mon[27627]: mon.b calling monitor election 2024-03-26T04:39:02.482 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:02 smithi114 ceph-mon[27627]: mon.a calling monitor election 2024-03-26T04:39:02.482 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:02 smithi114 ceph-mon[27627]: mon.c calling monitor election 2024-03-26T04:39:02.482 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:02 smithi114 ceph-mon[27627]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:39:02.482 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:02 smithi114 ceph-mon[27627]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:39:02.482 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:02 smithi114 ceph-mon[27627]: fsmap 2024-03-26T04:39:02.482 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:02 smithi114 ceph-mon[27627]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:39:02.482 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:02 smithi114 ceph-mon[27627]: mgrmap e13: y(active, since 63s) 2024-03-26T04:39:02.482 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:02 smithi114 ceph-mon[27627]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T04:39:02.482 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:02 smithi114 ceph-mon[27627]: Cluster is now healthy 2024-03-26T04:39:02.482 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:02 smithi114 ceph-mon[27627]: overall HEALTH_OK 2024-03-26T04:39:03.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:03 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:03.477 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:03 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:03.567 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:03 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:39:04.231 INFO:teuthology.orchestra.run.smithi038.stdout:# minimal ceph.conf for 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:39:04.231 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2024-03-26T04:39:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:39:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: mon_host = [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-03-26T04:39:04.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:04.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:39:04.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:39:04.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:39:04.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:04.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: Reconfiguring daemon mon.b on smithi114 2024-03-26T04:39:04.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.533 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:04.533 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.533 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:39:04.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:39:04.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:39:04.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:04.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: Reconfiguring daemon mon.b on smithi114 2024-03-26T04:39:04.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:04.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:39:04.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:39:04.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:39:04.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:04.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: Reconfiguring daemon mon.b on smithi114 2024-03-26T04:39:04.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:04.847 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T04:39:04.847 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:39:04.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:39:04.878 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:39:04.878 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:39:04.946 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-26T04:39:04.947 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:39:04.978 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-26T04:39:04.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:39:05.046 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-26T04:39:05.046 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:39:05.079 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-26T04:39:05.079 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:39:05.149 INFO:tasks.cephadm:Adding mgr.y on smithi038 2024-03-26T04:39:05.149 INFO:tasks.cephadm:Adding mgr.x on smithi114 2024-03-26T04:39:05.149 INFO:tasks.cephadm:Adding mgr.z on smithi177 2024-03-26T04:39:05.149 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi038=y;smithi114=x;smithi177=z' 2024-03-26T04:39:05.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:05 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/3437836905' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:05.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:05 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:05.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:05 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:05.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:05 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:39:05.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:05 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:39:05.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:05 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:05.430 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:39:05.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:05 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/3437836905' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:05.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:05 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:05.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:05 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:05.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:05 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:39:05.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:05 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:39:05.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:05 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:05.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:05 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/3437836905' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:05.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:05 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:05.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:05 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:05.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:05 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:39:05.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:05 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:39:05.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:05 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:06.472 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:06 smithi177 ceph-mon[27677]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:06.472 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:06 smithi177 ceph-mon[27677]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:39:06.472 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:06 smithi177 ceph-mon[27677]: Reconfiguring daemon mon.c on smithi177 2024-03-26T04:39:06.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:06 smithi038 ceph-mon[25836]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:06.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:06 smithi038 ceph-mon[25836]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:39:06.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:06 smithi038 ceph-mon[25836]: Reconfiguring daemon mon.c on smithi177 2024-03-26T04:39:06.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:06 smithi114 ceph-mon[27627]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:06.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:06 smithi114 ceph-mon[27627]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:39:06.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:06 smithi114 ceph-mon[27627]: Reconfiguring daemon mon.c on smithi177 2024-03-26T04:39:07.732 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mgr update... 2024-03-26T04:39:08.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:08 smithi177 ceph-mon[27677]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:08.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:08 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:08.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:08 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:08.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:08 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:08.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:08 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:08.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:08 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:08.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:08 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:08.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:08 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:08.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:08 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:39:08.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:08 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:39:08.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:08 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:39:08.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:08 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:08.394 DEBUG:teuthology.orchestra.run.smithi114:mgr.x> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T04:39:08.397 DEBUG:teuthology.orchestra.run.smithi177:mgr.z> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.z.service 2024-03-26T04:39:08.399 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T04:39:08.399 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:39:08.399 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:39:08.418 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T04:39:08.418 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/nvme1n1 2024-03-26T04:39:08.475 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/nvme1n1 2024-03-26T04:39:08.475 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:39:08.476 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-26T04:39:08.476 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:39:08.476 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:39:08.476 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-26 04:38:51.634575421 +0000 2024-03-26T04:39:08.476 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-26 04:35:53.283282735 +0000 2024-03-26T04:39:08.476 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-26 04:35:53.283282735 +0000 2024-03-26T04:39:08.476 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-26 04:35:53.274282917 +0000 2024-03-26T04:39:08.476 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-26T04:39:08.543 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:08 smithi038 ceph-mon[25836]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:08.543 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:08 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:08.543 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:08 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:08.543 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:08 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:08.544 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:08 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:08.544 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:08 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:08.544 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:08 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:08.544 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:08 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:08.544 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:08 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:39:08.544 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:08 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:39:08.544 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:08 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:39:08.544 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:08 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:08.547 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-26T04:39:08.547 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-26T04:39:08.547 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000314003 s, 1.6 MB/s 2024-03-26T04:39:08.548 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-26T04:39:08.605 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/nvme2n1 2024-03-26T04:39:08.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:08 smithi114 ceph-mon[27627]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:08.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:08 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:08.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:08 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:08.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:08 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:08.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:08 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:08.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:08 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:08.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:08 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:08.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:08 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:08.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:08 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:39:08.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:08 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:39:08.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:08 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:39:08.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:08 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:08.662 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/nvme2n1 2024-03-26T04:39:08.662 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:39:08.662 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-26T04:39:08.662 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:39:08.662 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:39:08.662 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-26 04:38:51.647575158 +0000 2024-03-26T04:39:08.662 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-26 04:35:53.433269347 +0000 2024-03-26T04:39:08.662 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-26 04:35:53.433269347 +0000 2024-03-26T04:39:08.662 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-26 04:35:53.420270897 +0000 2024-03-26T04:39:08.663 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-26T04:39:08.729 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-26T04:39:08.730 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-26T04:39:08.730 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000303079 s, 1.7 MB/s 2024-03-26T04:39:08.731 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-26T04:39:08.787 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/nvme3n1 2024-03-26T04:39:08.842 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/nvme3n1 2024-03-26T04:39:08.866 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:39:08.866 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-26T04:39:08.866 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:39:08.866 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:39:08.866 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-26 04:38:51.659574915 +0000 2024-03-26T04:39:08.866 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-26 04:35:53.584251344 +0000 2024-03-26T04:39:08.866 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-26 04:35:53.584251344 +0000 2024-03-26T04:39:08.866 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-26 04:35:53.575252417 +0000 2024-03-26T04:39:08.867 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-26T04:39:08.908 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-26T04:39:08.908 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-26T04:39:08.908 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000290794 s, 1.8 MB/s 2024-03-26T04:39:08.909 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-26T04:39:08.964 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/nvme4n1 2024-03-26T04:39:09.019 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/nvme4n1 2024-03-26T04:39:09.019 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:39:09.019 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-26T04:39:09.019 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:39:09.019 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:39:09.019 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-26 04:38:51.671574672 +0000 2024-03-26T04:39:09.019 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-26 04:35:53.760230360 +0000 2024-03-26T04:39:09.020 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-26 04:35:53.760230360 +0000 2024-03-26T04:39:09.020 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-26 04:35:53.751231434 +0000 2024-03-26T04:39:09.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-26T04:39:09.085 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-26T04:39:09.085 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-26T04:39:09.085 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000301866 s, 1.7 MB/s 2024-03-26T04:39:09.086 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-26T04:39:09.142 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-26T04:39:09.142 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:39:09.160 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T04:39:09.161 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/nvme1n1 2024-03-26T04:39:09.218 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/nvme1n1 2024-03-26T04:39:09.218 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:39:09.219 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-26T04:39:09.219 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:39:09.219 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:39:09.219 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-26 04:39:02.424672448 +0000 2024-03-26T04:39:09.219 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-26 04:35:54.875310269 +0000 2024-03-26T04:39:09.219 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-26 04:35:54.875310269 +0000 2024-03-26T04:39:09.219 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-26 04:35:54.863310311 +0000 2024-03-26T04:39:09.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-26T04:39:09.288 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-26T04:39:09.288 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-26T04:39:09.288 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000227341 s, 2.3 MB/s 2024-03-26T04:39:09.289 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-26T04:39:09.347 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/nvme2n1 2024-03-26T04:39:09.405 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/nvme2n1 2024-03-26T04:39:09.405 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:39:09.405 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-26T04:39:09.405 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:39:09.405 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:39:09.405 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-26 04:39:02.434672414 +0000 2024-03-26T04:39:09.405 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-26 04:35:55.014309782 +0000 2024-03-26T04:39:09.405 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-26 04:35:55.014309782 +0000 2024-03-26T04:39:09.405 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-26 04:35:55.009309800 +0000 2024-03-26T04:39:09.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-26T04:39:09.470 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:09 smithi114 ceph-mon[27627]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=y;smithi114=x;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:09.470 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:09 smithi114 ceph-mon[27627]: Saving service mgr spec with placement smithi038=y;smithi114=x;smithi177=z;count:3 2024-03-26T04:39:09.470 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:09 smithi114 ceph-mon[27627]: Deploying daemon mgr.z on smithi177 2024-03-26T04:39:09.473 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-26T04:39:09.473 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-26T04:39:09.473 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.00023738 s, 2.2 MB/s 2024-03-26T04:39:09.474 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-26T04:39:09.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:09 smithi038 ceph-mon[25836]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=y;smithi114=x;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:09.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:09 smithi038 ceph-mon[25836]: Saving service mgr spec with placement smithi038=y;smithi114=x;smithi177=z;count:3 2024-03-26T04:39:09.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:09 smithi038 ceph-mon[25836]: Deploying daemon mgr.z on smithi177 2024-03-26T04:39:09.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:09 smithi177 ceph-mon[27677]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=y;smithi114=x;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:09.530 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:09 smithi177 ceph-mon[27677]: Saving service mgr spec with placement smithi038=y;smithi114=x;smithi177=z;count:3 2024-03-26T04:39:09.530 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:09 smithi177 ceph-mon[27677]: Deploying daemon mgr.z on smithi177 2024-03-26T04:39:09.532 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/nvme3n1 2024-03-26T04:39:09.590 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/nvme3n1 2024-03-26T04:39:09.590 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:39:09.590 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-26T04:39:09.591 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:39:09.591 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:39:09.591 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-26 04:39:02.445672377 +0000 2024-03-26T04:39:09.591 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-26 04:35:55.164309258 +0000 2024-03-26T04:39:09.591 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-26 04:35:55.164309258 +0000 2024-03-26T04:39:09.591 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-26 04:35:55.153309295 +0000 2024-03-26T04:39:09.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-26T04:39:09.657 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-26T04:39:09.658 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-26T04:39:09.658 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000218835 s, 2.3 MB/s 2024-03-26T04:39:09.659 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-26T04:39:09.715 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/nvme4n1 2024-03-26T04:39:09.770 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/nvme4n1 2024-03-26T04:39:09.770 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:39:09.770 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-26T04:39:09.770 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:39:09.770 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:39:09.770 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-26 04:39:02.456672339 +0000 2024-03-26T04:39:09.770 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-26 04:35:55.310308763 +0000 2024-03-26T04:39:09.770 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-26 04:35:55.310308763 +0000 2024-03-26T04:39:09.770 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-26 04:35:55.302308790 +0000 2024-03-26T04:39:09.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-26T04:39:09.835 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-26T04:39:09.836 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-26T04:39:09.836 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000216932 s, 2.4 MB/s 2024-03-26T04:39:09.836 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-26T04:39:09.892 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-26T04:39:09.892 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:39:09.908 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T04:39:09.908 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/nvme1n1 2024-03-26T04:39:09.965 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/nvme1n1 2024-03-26T04:39:09.965 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:39:09.966 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-26T04:39:09.966 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:39:09.966 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:39:09.966 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-26 04:39:02.828379332 +0000 2024-03-26T04:39:09.966 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-26 04:35:56.514645652 +0000 2024-03-26T04:39:09.966 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-26 04:35:56.514645652 +0000 2024-03-26T04:39:09.966 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-26 04:35:56.503645962 +0000 2024-03-26T04:39:09.966 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-26T04:39:10.036 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-26T04:39:10.036 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-26T04:39:10.036 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000270308 s, 1.9 MB/s 2024-03-26T04:39:10.037 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-26T04:39:10.093 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/nvme2n1 2024-03-26T04:39:10.148 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/nvme2n1 2024-03-26T04:39:10.149 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:39:10.149 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-26T04:39:10.149 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:39:10.149 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:39:10.149 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-26 04:39:02.838379049 +0000 2024-03-26T04:39:10.149 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-26 04:35:56.658641582 +0000 2024-03-26T04:39:10.149 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-26 04:35:56.658641582 +0000 2024-03-26T04:39:10.149 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-26 04:35:56.647641893 +0000 2024-03-26T04:39:10.149 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-26T04:39:10.215 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-26T04:39:10.215 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-26T04:39:10.215 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000213563 s, 2.4 MB/s 2024-03-26T04:39:10.216 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-26T04:39:10.273 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/nvme3n1 2024-03-26T04:39:10.328 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/nvme3n1 2024-03-26T04:39:10.328 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:39:10.328 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-26T04:39:10.328 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:39:10.329 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:39:10.329 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-26 04:39:02.849378738 +0000 2024-03-26T04:39:10.329 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-26 04:35:56.802637512 +0000 2024-03-26T04:39:10.329 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-26 04:35:56.802637512 +0000 2024-03-26T04:39:10.329 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-26 04:35:56.791637823 +0000 2024-03-26T04:39:10.329 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-26T04:39:10.392 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:10 smithi177 ceph-mon[27677]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:10.394 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-26T04:39:10.394 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-26T04:39:10.395 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000212297 s, 2.4 MB/s 2024-03-26T04:39:10.396 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-26T04:39:10.451 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/nvme4n1 2024-03-26T04:39:10.508 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/nvme4n1 2024-03-26T04:39:10.508 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:39:10.508 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-26T04:39:10.508 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:39:10.508 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:39:10.508 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-26 04:39:02.859378455 +0000 2024-03-26T04:39:10.508 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-26 04:35:56.946633443 +0000 2024-03-26T04:39:10.508 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-26 04:35:56.946633443 +0000 2024-03-26T04:39:10.508 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-26 04:35:56.935633754 +0000 2024-03-26T04:39:10.508 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-26T04:39:10.540 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:10 smithi038 ceph-mon[25836]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:10.576 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-26T04:39:10.576 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-26T04:39:10.576 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000203304 s, 2.5 MB/s 2024-03-26T04:39:10.577 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-26T04:39:10.634 INFO:tasks.cephadm:Deploying osd.0 on smithi038 with /dev/nvme4n1... 2024-03-26T04:39:10.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-26T04:39:10.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:10 smithi114 ceph-mon[27627]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:10.979 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:39:11.399 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:11.076+0000 7f29dccf9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:39:11.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:11 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:11 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:11 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:11 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:39:11.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:11 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:39:11.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:11 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:39:11.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:11 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:11.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:11 smithi038 ceph-mon[25836]: Deploying daemon mgr.x on smithi114 2024-03-26T04:39:11.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:11 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:11 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:11 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:11 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:11 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:11 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:11 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:39:11.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:11 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:39:11.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:11 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:39:11.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:11 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:11.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:11 smithi114 ceph-mon[27627]: Deploying daemon mgr.x on smithi114 2024-03-26T04:39:11.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:11 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:11 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:11 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.899 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:11.720+0000 7f29dccf9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:39:11.899 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:11.811+0000 7f29dccf9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:39:11.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:39:11.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:39:11.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:39:11.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:11.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-mon[27677]: Deploying daemon mgr.x on smithi114 2024-03-26T04:39:11.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:11.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:12.389 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:11 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:11.953+0000 7f29dccf9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:39:12.390 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:12 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:12.046+0000 7f29dccf9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:39:12.390 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:12 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:12.128+0000 7f29dccf9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:39:12.390 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:12 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:12.389+0000 7f29dccf9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:39:12.649 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:12 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:12.498+0000 7f29dccf9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:39:12.649 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:12 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:12.583+0000 7f29dccf9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:39:12.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:12 smithi038 ceph-mon[25836]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:12.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:12 smithi114 ceph-mon[27627]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:12.896 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:12 smithi114 podman[29277]: 2024-03-26 04:39:12.75802748 +0000 UTC m=+0.017598943 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:39:13.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:12 smithi177 ceph-mon[27677]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:13.149 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:12 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:12.677+0000 7f29dccf9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:39:13.149 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:12 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:12.854+0000 7f29dccf9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:39:13.395 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:13 smithi114 podman[29277]: 2024-03-26 04:39:13.056705606 +0000 UTC m=+0.316277052 container init 5b35811f1de0cd99bd8d3f5759bc035cb35572bb0477c15c545490852de5bd83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, ceph=True, GIT_BRANCH=HEAD) 2024-03-26T04:39:13.395 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:13 smithi114 podman[29277]: 2024-03-26 04:39:13.062241361 +0000 UTC m=+0.321812808 container start 5b35811f1de0cd99bd8d3f5759bc035cb35572bb0477c15c545490852de5bd83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-26T04:39:13.395 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:13 smithi114 ceph-mgr[29315]: -- 172.21.15.114:0/3386411980 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556f2f9ce1a0 con 0x556f2f9a4800 2024-03-26T04:39:13.396 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:13 smithi114 bash[29277]: 5b35811f1de0cd99bd8d3f5759bc035cb35572bb0477c15c545490852de5bd83 2024-03-26T04:39:13.396 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:13 smithi114 systemd[1]: Started Ceph mgr.x for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:39:13.479 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/nvme4n1 2024-03-26T04:39:13.479 INFO:teuthology.orchestra.run.smithi038.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-26T04:39:13.479 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-26T04:39:13.480 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-03-26T04:39:13.480 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-03-26T04:39:13.480 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0174693 s, 600 MB/s 2024-03-26T04:39:13.480 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-03-26T04:39:13.726 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:13 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:13.629+0000 7f9572bed200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:39:13.726 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:13 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:13.726+0000 7f9572bed200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:39:14.149 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:13.824+0000 7f29dccf9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:39:14.149 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:13.952+0000 7f29dccf9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:39:14.164 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi038:/dev/nvme4n1 2024-03-26T04:39:14.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:14.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:14.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:14.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:14.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:39:14.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:39:14.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:39:14.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:39:14.366 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:14.366 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: Reconfiguring daemon mgr.y on smithi038 2024-03-26T04:39:14.366 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.426 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:14.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:14.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:14.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:14.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:39:14.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:39:14.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:39:14.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:39:14.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:14.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: Reconfiguring daemon mgr.y on smithi038 2024-03-26T04:39:14.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.565 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:14.287+0000 7f29dccf9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:39:14.565 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:14.374+0000 7f29dccf9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:39:14.565 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:14.565+0000 7f29dccf9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:39:14.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:14.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: Reconfiguring daemon mgr.y on smithi038 2024-03-26T04:39:14.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:14.645 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:14.422+0000 7f9572bed200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:39:14.645 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:14.529+0000 7f9572bed200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:39:14.861 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:39:14.899 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:14 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:14.747+0000 7f29dccf9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:39:15.017 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:14.672+0000 7f9572bed200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:39:15.017 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:14.768+0000 7f9572bed200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:39:15.017 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:14 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:14.852+0000 7f9572bed200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:39:15.287 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:15 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:15.027+0000 7f29dccf9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:39:15.287 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:15 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:15.287+0000 7f29dccf9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:39:15.290 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:15 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:15.122+0000 7f9572bed200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:39:15.290 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:15 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:15.237+0000 7f9572bed200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:39:15.615 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:15 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:15.326+0000 7f9572bed200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:39:15.615 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:15 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:15.423+0000 7f9572bed200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:39:15.615 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:15 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:15.614+0000 7f9572bed200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:39:15.649 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:15 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:15.519+0000 7f29dccf9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:39:16.468 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:16 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:16.348+0000 7f29dccf9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:16 smithi114 ceph-mon[27627]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:16 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:16 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:16 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:16 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:16.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:16 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:16.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:16 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:16.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:16 smithi114 ceph-mon[27627]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:16 smithi038 ceph-mon[25836]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:16 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:16 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:16 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:16 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:16 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:16 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:16 smithi038 ceph-mon[25836]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:39:16.826 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:16 smithi177 ceph-mon[27677]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:16.826 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:16 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:16.826 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:16 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:16.826 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:16 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:16.826 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:16 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:16.826 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:16 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:16.826 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:16 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:16.827 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:16 smithi177 ceph-mon[27677]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:39:17.089 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:16 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:16.646+0000 7f9572bed200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:39:17.089 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:16 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:16.782+0000 7f9572bed200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:39:17.111 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:16 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:16.949+0000 7f29dccf9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:39:17.112 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:17.047+0000 7f29dccf9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:39:17.395 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:17.122+0000 7f9572bed200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:39:17.395 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:17.210+0000 7f9572bed200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:39:17.399 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:39:17.287+0000 7f29dccf9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:39:17.741 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:17.404+0000 7f9572bed200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:39:17.742 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:17.587+0000 7f9572bed200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:39:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:39:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:39:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: Standby manager daemon z started 2024-03-26T04:39:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.177:0/455020583' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T04:39:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.177:0/455020583' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:39:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.177:0/455020583' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T04:39:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:17 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.177:0/455020583' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:39:18.061 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:18.061 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:18.061 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:18.062 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:18.062 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:18.062 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:18.062 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:39:18.062 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:39:18.062 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:18.062 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: Standby manager daemon z started 2024-03-26T04:39:18.062 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.177:0/455020583' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T04:39:18.062 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.177:0/455020583' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:39:18.062 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.177:0/455020583' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T04:39:18.062 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:17 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.177:0/455020583' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:39:18.126 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:17.868+0000 7f9572bed200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:39:18.126 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:18 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:18.126+0000 7f9572bed200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:39:18.126 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:18.126 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:18.127 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:18.127 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:18.127 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:18.127 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:18.127 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:39:18.127 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:39:18.127 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:18.127 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: Standby manager daemon z started 2024-03-26T04:39:18.127 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.177:0/455020583' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T04:39:18.127 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.177:0/455020583' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:39:18.127 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.177:0/455020583' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T04:39:18.127 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:17 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.177:0/455020583' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:39:18.395 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:18 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:18.359+0000 7f9572bed200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:39:19.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:19 smithi038 ceph-mon[25836]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-26T04:39:19.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:19 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T04:39:19.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:19 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:19 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:19 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:19.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:19 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:19 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:19.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:19 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:19.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:19 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:19 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:19 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:19 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.395 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:19.162+0000 7f9572bed200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:39:19.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-mon[27627]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-26T04:39:19.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T04:39:19.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:19.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:19.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:19.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:19 smithi177 ceph-mon[27677]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-26T04:39:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:19 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T04:39:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:19 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:19 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:19 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:19 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:19 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:19 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:19 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:19 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:19 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:19.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:19 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:20.033 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:19.723+0000 7f9572bed200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:39:20.033 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:19 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:19.819+0000 7f9572bed200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:39:20.033 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:39:20 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:39:20.032+0000 7f9572bed200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:39:20.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:20 smithi038 ceph-mon[25836]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:20.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:20 smithi038 ceph-mon[25836]: Standby manager daemon x started 2024-03-26T04:39:20.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:20 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.114:0/3553110292' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:39:20.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:20 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.114:0/3553110292' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:39:20.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:20 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.114:0/3553110292' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:39:20.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:20 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.114:0/3553110292' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:39:20.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:20 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/704672220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36825eb5-ff6e-4e19-adab-791266b00c62"}]: dispatch 2024-03-26T04:39:20.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:20 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/704672220' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "36825eb5-ff6e-4e19-adab-791266b00c62"}]': finished 2024-03-26T04:39:20.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:20 smithi038 ceph-mon[25836]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:39:20.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:20 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:20.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:20 smithi177 ceph-mon[27677]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:20.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:20 smithi177 ceph-mon[27677]: Standby manager daemon x started 2024-03-26T04:39:20.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:20 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.114:0/3553110292' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:39:20.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:20 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.114:0/3553110292' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:39:20.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:20 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.114:0/3553110292' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:39:20.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:20 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.114:0/3553110292' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:39:20.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:20 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/704672220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36825eb5-ff6e-4e19-adab-791266b00c62"}]: dispatch 2024-03-26T04:39:20.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:20 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/704672220' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "36825eb5-ff6e-4e19-adab-791266b00c62"}]': finished 2024-03-26T04:39:20.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:20 smithi177 ceph-mon[27677]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:39:20.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:20 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:20.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:20 smithi114 ceph-mon[27627]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:20.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:20 smithi114 ceph-mon[27627]: Standby manager daemon x started 2024-03-26T04:39:20.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:20 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.114:0/3553110292' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:39:20.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:20 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.114:0/3553110292' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:39:20.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:20 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.114:0/3553110292' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:39:20.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:20 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.114:0/3553110292' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:39:20.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:20 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/704672220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36825eb5-ff6e-4e19-adab-791266b00c62"}]: dispatch 2024-03-26T04:39:20.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:20 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/704672220' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "36825eb5-ff6e-4e19-adab-791266b00c62"}]': finished 2024-03-26T04:39:20.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:20 smithi114 ceph-mon[27627]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:39:20.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:20 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:21.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:21 smithi038 ceph-mon[25836]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-26T04:39:21.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:21 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:39:21.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:21 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/3962380302' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:39:21.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:21 smithi114 ceph-mon[27627]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-26T04:39:21.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:21 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:39:21.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:21 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/3962380302' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:39:21.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:21 smithi177 ceph-mon[27677]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-26T04:39:21.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:21 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:39:21.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:21 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/3962380302' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:39:22.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:22 smithi038 ceph-mon[25836]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:22.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:22 smithi114 ceph-mon[27627]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:22.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:22 smithi177 ceph-mon[27677]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:24.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:24 smithi038 ceph-mon[25836]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:24.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:24 smithi114 ceph-mon[27627]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:24.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:24 smithi177 ceph-mon[27677]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:26.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:26 smithi038 ceph-mon[25836]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:26.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:26 smithi114 ceph-mon[27627]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:26.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:26 smithi177 ceph-mon[27677]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:27.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:27 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:39:27.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:27 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:27.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:27 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:39:27.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:27 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:27.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:27 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:39:27.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:27 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:28.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:28 smithi038 ceph-mon[25836]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:28.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:28 smithi038 ceph-mon[25836]: Deploying daemon osd.0 on smithi038 2024-03-26T04:39:28.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:28 smithi114 ceph-mon[27627]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:28.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:28 smithi114 ceph-mon[27627]: Deploying daemon osd.0 on smithi038 2024-03-26T04:39:28.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:28 smithi177 ceph-mon[27677]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:28.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:28 smithi177 ceph-mon[27677]: Deploying daemon osd.0 on smithi038 2024-03-26T04:39:30.676 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:30 smithi038 ceph-mon[25836]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:30.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:30 smithi114 ceph-mon[27627]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:30.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:30 smithi177 ceph-mon[27677]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:32.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:32 smithi038 ceph-mon[25836]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:32.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:32 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:32.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:32 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:32.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:32 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:32.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:32 smithi114 ceph-mon[27627]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:32.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:32 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:32.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:32 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:32.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:32 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:32.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:32 smithi177 ceph-mon[27677]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:32.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:32 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:32.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:32 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:32.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:32 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:34.348 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:34 smithi038 ceph-mon[25836]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:34.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:34 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:34.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:34 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:34.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:34 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:34.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:34 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:34.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:34 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:34.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:34 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:34.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:34 smithi114 ceph-mon[27627]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:34.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:34 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:34.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:34 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:34.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:34 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:34.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:34 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:34.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:34 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:34.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:34 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:34.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:34 smithi177 ceph-mon[27677]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:34.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:34 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:34.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:34 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:34.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:34 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:34.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:34 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:34.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:34 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:34.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:34 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:36.537 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:36 smithi114 ceph-mon[27627]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:36.537 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:36 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:36.537 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:36 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:36.537 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:36 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:36.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:36 smithi038 ceph-mon[25836]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:36.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:36.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:36.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:36.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:36 smithi177 ceph-mon[27677]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:36.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:36 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:36.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:36 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:36.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:36 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:37.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:37 smithi177 ceph-mon[27677]: from='osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:39:37.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:37 smithi038 ceph-mon[25836]: from='osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:39:37.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:37 smithi114 ceph-mon[27627]: from='osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:39:38.050 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 0 on host 'smithi038' 2024-03-26T04:39:38.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:38.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:39:38.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:39:38.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T04:39:38.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:38.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:38.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:39:38.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:38.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:38.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:38.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:39:38.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:39:38.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T04:39:38.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:38.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:38.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:39:38.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:38.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:38.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.642 DEBUG:teuthology.orchestra.run.smithi038:osd.0> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T04:39:38.644 INFO:tasks.cephadm:Deploying osd.1 on smithi114 with /dev/nvme4n1... 2024-03-26T04:39:38.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-26T04:39:38.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:38.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:39:38.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:39:38.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T04:39:38.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:38.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:38.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:39:38.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:38.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:38.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:38.891 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:39:39.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:39 smithi038 ceph-mon[25836]: Adjusting osd_memory_target on smithi038 to 16069M 2024-03-26T04:39:39.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:39 smithi038 ceph-mon[25836]: from='osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T04:39:39.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:39 smithi038 ceph-mon[25836]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:39:39.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:39 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:39.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:39 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:39.622 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:39 smithi177 ceph-mon[27677]: Adjusting osd_memory_target on smithi038 to 16069M 2024-03-26T04:39:39.622 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:39 smithi177 ceph-mon[27677]: from='osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T04:39:39.622 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:39 smithi177 ceph-mon[27677]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:39:39.622 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:39 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:39.622 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:39 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:39.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:39 smithi114 ceph-mon[27627]: Adjusting osd_memory_target on smithi038 to 16069M 2024-03-26T04:39:39.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:39 smithi114 ceph-mon[27627]: from='osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T04:39:39.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:39 smithi114 ceph-mon[27627]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:39:39.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:39 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:39.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:39 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:40.356 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:40 smithi114 ceph-mon[27627]: purged_snaps scrub starts 2024-03-26T04:39:40.356 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:40 smithi114 ceph-mon[27627]: purged_snaps scrub ok 2024-03-26T04:39:40.356 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:40 smithi114 ceph-mon[27627]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:40.356 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:40 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:40.356 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:40 smithi114 ceph-mon[27627]: osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714] boot 2024-03-26T04:39:40.356 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:40 smithi114 ceph-mon[27627]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:39:40.356 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:40 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:40.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:40 smithi177 ceph-mon[27677]: purged_snaps scrub starts 2024-03-26T04:39:40.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:40 smithi177 ceph-mon[27677]: purged_snaps scrub ok 2024-03-26T04:39:40.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:40 smithi177 ceph-mon[27677]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:40.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:40 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:40.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:40 smithi177 ceph-mon[27677]: osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714] boot 2024-03-26T04:39:40.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:40 smithi177 ceph-mon[27677]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:39:40.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:40 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:40.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:40 smithi038 ceph-mon[25836]: purged_snaps scrub starts 2024-03-26T04:39:40.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:40 smithi038 ceph-mon[25836]: purged_snaps scrub ok 2024-03-26T04:39:40.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:40 smithi038 ceph-mon[25836]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:39:40.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:40 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:40.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:40 smithi038 ceph-mon[25836]: osd.0 [v2:172.21.15.38:6802/755874714,v1:172.21.15.38:6803/755874714] boot 2024-03-26T04:39:40.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:40 smithi038 ceph-mon[25836]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:39:40.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:40 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:39:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/nvme4n1 2024-03-26T04:39:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-26T04:39:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-26T04:39:40.791 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-03-26T04:39:40.792 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-03-26T04:39:40.792 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125466 s, 836 MB/s 2024-03-26T04:39:40.792 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-03-26T04:39:41.295 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:41 smithi114 ceph-mon[27627]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:39:41.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi114:/dev/nvme4n1 2024-03-26T04:39:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:39:41.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:41 smithi038 ceph-mon[25836]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:39:41.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:41 smithi177 ceph-mon[27677]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:39:42.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:42 smithi038 ceph-mon[25836]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:42.616 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:42 smithi114 ceph-mon[27627]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:42.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:42 smithi177 ceph-mon[27677]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:44.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:44 smithi114 ceph-mon[27627]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:44.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:44 smithi114 ceph-mon[27627]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:44.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:44 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:39:44.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:44 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:39:44.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:44 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:44.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:44 smithi038 ceph-mon[25836]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:44.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:44 smithi038 ceph-mon[25836]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:44.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:44 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:39:44.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:44 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:39:44.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:44 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:44.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:44 smithi177 ceph-mon[27677]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:44.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:44 smithi177 ceph-mon[27677]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:44.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:44 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:39:44.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:44 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:39:44.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:44 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:46.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:46 smithi038 ceph-mon[25836]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:46.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:46 smithi038 ceph-mon[25836]: from='client.? 172.21.15.114:0/1569088818' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "631b8290-ee85-4525-be34-13e99b5e9573"}]: dispatch 2024-03-26T04:39:46.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:46 smithi038 ceph-mon[25836]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "631b8290-ee85-4525-be34-13e99b5e9573"}]: dispatch 2024-03-26T04:39:46.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:46 smithi038 ceph-mon[25836]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "631b8290-ee85-4525-be34-13e99b5e9573"}]': finished 2024-03-26T04:39:46.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:46 smithi038 ceph-mon[25836]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:39:46.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:46 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:39:46.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:46 smithi114 ceph-mon[27627]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:46.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:46 smithi114 ceph-mon[27627]: from='client.? 172.21.15.114:0/1569088818' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "631b8290-ee85-4525-be34-13e99b5e9573"}]: dispatch 2024-03-26T04:39:46.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:46 smithi114 ceph-mon[27627]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "631b8290-ee85-4525-be34-13e99b5e9573"}]: dispatch 2024-03-26T04:39:46.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:46 smithi114 ceph-mon[27627]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "631b8290-ee85-4525-be34-13e99b5e9573"}]': finished 2024-03-26T04:39:46.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:46 smithi114 ceph-mon[27627]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:39:46.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:46 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:39:46.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:46 smithi177 ceph-mon[27677]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:46.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:46 smithi177 ceph-mon[27677]: from='client.? 172.21.15.114:0/1569088818' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "631b8290-ee85-4525-be34-13e99b5e9573"}]: dispatch 2024-03-26T04:39:46.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:46 smithi177 ceph-mon[27677]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "631b8290-ee85-4525-be34-13e99b5e9573"}]: dispatch 2024-03-26T04:39:46.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:46 smithi177 ceph-mon[27677]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "631b8290-ee85-4525-be34-13e99b5e9573"}]': finished 2024-03-26T04:39:46.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:46 smithi177 ceph-mon[27677]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:39:46.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:46 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:39:47.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:47 smithi038 ceph-mon[25836]: from='client.? 172.21.15.114:0/2463564729' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:39:47.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:47 smithi114 ceph-mon[27627]: from='client.? 172.21.15.114:0/2463564729' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:39:47.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:47 smithi177 ceph-mon[27677]: from='client.? 172.21.15.114:0/2463564729' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:39:48.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:48 smithi038 ceph-mon[25836]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:48.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:48 smithi114 ceph-mon[27627]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:48.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:48 smithi177 ceph-mon[27677]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:50.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:50 smithi038 ceph-mon[25836]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:50.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:50.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:50.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:50 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:50.618 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:50 smithi114 ceph-mon[27627]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:50.618 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:50 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:50.618 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:50 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:50.618 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:50 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:50.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:50 smithi177 ceph-mon[27677]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:50.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:50 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:50.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:50 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:50.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:50 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:51.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:51 smithi114 ceph-mon[27627]: Detected new or changed devices on smithi038 2024-03-26T04:39:51.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:51 smithi038 ceph-mon[25836]: Detected new or changed devices on smithi038 2024-03-26T04:39:51.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:51 smithi177 ceph-mon[27677]: Detected new or changed devices on smithi038 2024-03-26T04:39:52.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:52 smithi038 ceph-mon[25836]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:52.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:52 smithi114 ceph-mon[27627]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:52.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:52 smithi177 ceph-mon[27677]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:53.409 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:53 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:39:53.410 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:53 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:53.410 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:53 smithi114 ceph-mon[27627]: Deploying daemon osd.1 on smithi114 2024-03-26T04:39:53.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:53 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:39:53.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:53 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:53.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:53 smithi038 ceph-mon[25836]: Deploying daemon osd.1 on smithi114 2024-03-26T04:39:53.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:53 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:39:53.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:53 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:53.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:53 smithi177 ceph-mon[27677]: Deploying daemon osd.1 on smithi114 2024-03-26T04:39:54.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:54 smithi114 ceph-mon[27627]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:54.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:54 smithi038 ceph-mon[25836]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:54.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:54 smithi177 ceph-mon[27677]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:56.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:56 smithi038 ceph-mon[25836]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:56.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:56 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:56.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:56 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:56.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:56 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:56.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:56 smithi114 ceph-mon[27627]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:56.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:56 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:56.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:56 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:56.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:56 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:56.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:56 smithi177 ceph-mon[27677]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:56.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:56 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:56.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:56 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:56.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:56 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:58.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:58 smithi114 ceph-mon[27627]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:58.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:58 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:58.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:58 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:58.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:58 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:58.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:58 smithi177 ceph-mon[27677]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:58.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:58 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:58.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:58 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:58.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:58 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:58.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:58 smithi038 ceph-mon[25836]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:39:58.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:58 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:58.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:58 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:58.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:58 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:59.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:59 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:59.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:59 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:59.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:59 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:59.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:59 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:59.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:59 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:59.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:39:59 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:59.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:59 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:59.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:59 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:59.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:59 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:59.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:59 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:59.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:59 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:59.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:39:59 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:59.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:59 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:59.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:59 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:39:59.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:59 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:59.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:59 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:59.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:59 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:59.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:39:59 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:00.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:00 smithi038 ceph-mon[25836]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:40:00.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:00 smithi038 ceph-mon[25836]: overall HEALTH_OK 2024-03-26T04:40:00.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:00 smithi114 ceph-mon[27627]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:40:00.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:00 smithi114 ceph-mon[27627]: overall HEALTH_OK 2024-03-26T04:40:00.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:00 smithi177 ceph-mon[27677]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:40:00.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:00 smithi177 ceph-mon[27677]: overall HEALTH_OK 2024-03-26T04:40:02.466 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 1 on host 'smithi114' 2024-03-26T04:40:02.756 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:02 smithi114 ceph-mon[27627]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:40:02.757 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:02 smithi114 ceph-mon[27627]: from='osd.1 [v2:172.21.15.114:6800/3929328856,v1:172.21.15.114:6801/3929328856]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:40:02.757 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:02 smithi114 ceph-mon[27627]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:40:02.757 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:02 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:02.757 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:02 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:02.757 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:02 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:02.757 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:02 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:02.757 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:02 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:02.757 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:02 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:02.757 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:02 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:02.757 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:02 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:02.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:02 smithi177 ceph-mon[27677]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:40:02.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:02 smithi177 ceph-mon[27677]: from='osd.1 [v2:172.21.15.114:6800/3929328856,v1:172.21.15.114:6801/3929328856]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:40:02.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:02 smithi177 ceph-mon[27677]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:40:02.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:02 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:02.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:02 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:02.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:02 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:02.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:02 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:02.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:02 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:02.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:02 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:02.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:02 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:02.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:02 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:03.025 DEBUG:teuthology.orchestra.run.smithi114:osd.1> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T04:40:03.028 INFO:tasks.cephadm:Deploying osd.2 on smithi177 with /dev/nvme4n1... 2024-03-26T04:40:03.028 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-26T04:40:03.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:02 smithi038 ceph-mon[25836]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:40:03.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:02 smithi038 ceph-mon[25836]: from='osd.1 [v2:172.21.15.114:6800/3929328856,v1:172.21.15.114:6801/3929328856]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:40:03.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:02 smithi038 ceph-mon[25836]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:40:03.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:02 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:03.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:02 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:03.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:02 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:03.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:02 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:03.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:02 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:03.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:02 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:03.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:02 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:03.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:02 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:03.277 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:40:03.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:03 smithi177 ceph-mon[27677]: Adjusting osd_memory_target on smithi114 to 16064M 2024-03-26T04:40:03.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:03 smithi177 ceph-mon[27677]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:40:03.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:03 smithi177 ceph-mon[27677]: from='osd.1 [v2:172.21.15.114:6800/3929328856,v1:172.21.15.114:6801/3929328856]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-26T04:40:03.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:03 smithi177 ceph-mon[27677]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:40:03.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:03 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:03.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:03 smithi177 ceph-mon[27677]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-26T04:40:03.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:03 smithi177 ceph-mon[27677]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-26T04:40:03.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:03 smithi177 ceph-mon[27677]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:40:03.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:03 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:03.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:03 smithi114 ceph-mon[27627]: Adjusting osd_memory_target on smithi114 to 16064M 2024-03-26T04:40:03.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:03 smithi114 ceph-mon[27627]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:40:03.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:03 smithi114 ceph-mon[27627]: from='osd.1 [v2:172.21.15.114:6800/3929328856,v1:172.21.15.114:6801/3929328856]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-26T04:40:03.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:03 smithi114 ceph-mon[27627]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:40:03.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:03 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:03.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:03 smithi114 ceph-mon[27627]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-26T04:40:03.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:03 smithi114 ceph-mon[27627]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-26T04:40:03.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:03 smithi114 ceph-mon[27627]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:40:03.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:03 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:04.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:03 smithi038 ceph-mon[25836]: Adjusting osd_memory_target on smithi114 to 16064M 2024-03-26T04:40:04.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:03 smithi038 ceph-mon[25836]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:40:04.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:03 smithi038 ceph-mon[25836]: from='osd.1 [v2:172.21.15.114:6800/3929328856,v1:172.21.15.114:6801/3929328856]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-26T04:40:04.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:03 smithi038 ceph-mon[25836]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:40:04.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:03 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:04.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:03 smithi038 ceph-mon[25836]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-26T04:40:04.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:03 smithi038 ceph-mon[25836]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-26T04:40:04.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:03 smithi038 ceph-mon[25836]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:40:04.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:03 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:04.396 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:40:04 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1[32903]: 2024-03-26T04:40:04.093+0000 7f60d13d7700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T04:40:04.396 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:40:04 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1[32903]: 2024-03-26T04:40:04.095+0000 7f60c89c1700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:40:04.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:04 smithi114 ceph-mon[27627]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:40:04.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:04.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:04 smithi114 ceph-mon[27627]: from='osd.1 ' entity='osd.1' 2024-03-26T04:40:04.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:04 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:04.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:04 smithi177 ceph-mon[27677]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:40:04.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:04.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:04 smithi177 ceph-mon[27677]: from='osd.1 ' entity='osd.1' 2024-03-26T04:40:04.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:04 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:05.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:04 smithi038 ceph-mon[25836]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:40:05.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:05.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:04 smithi038 ceph-mon[25836]: from='osd.1 ' entity='osd.1' 2024-03-26T04:40:05.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:04 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:05.103 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/nvme4n1 2024-03-26T04:40:05.103 INFO:teuthology.orchestra.run.smithi177.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-26T04:40:05.103 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-26T04:40:05.103 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-26T04:40:05.103 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-26T04:40:05.103 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125394 s, 836 MB/s 2024-03-26T04:40:05.103 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-26T04:40:05.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:05 smithi177 ceph-mon[27677]: purged_snaps scrub starts 2024-03-26T04:40:05.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:05 smithi177 ceph-mon[27677]: purged_snaps scrub ok 2024-03-26T04:40:05.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:05 smithi177 ceph-mon[27677]: osd.1 [v2:172.21.15.114:6800/3929328856,v1:172.21.15.114:6801/3929328856] boot 2024-03-26T04:40:05.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:05 smithi177 ceph-mon[27677]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:40:05.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:05 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:05.711 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi177:/dev/nvme4n1 2024-03-26T04:40:05.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:05 smithi114 ceph-mon[27627]: purged_snaps scrub starts 2024-03-26T04:40:05.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:05 smithi114 ceph-mon[27627]: purged_snaps scrub ok 2024-03-26T04:40:05.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:05 smithi114 ceph-mon[27627]: osd.1 [v2:172.21.15.114:6800/3929328856,v1:172.21.15.114:6801/3929328856] boot 2024-03-26T04:40:05.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:05 smithi114 ceph-mon[27627]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:40:05.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:05 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:05.944 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:40:06.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:05 smithi038 ceph-mon[25836]: purged_snaps scrub starts 2024-03-26T04:40:06.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:05 smithi038 ceph-mon[25836]: purged_snaps scrub ok 2024-03-26T04:40:06.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:05 smithi038 ceph-mon[25836]: osd.1 [v2:172.21.15.114:6800/3929328856,v1:172.21.15.114:6801/3929328856] boot 2024-03-26T04:40:06.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:05 smithi038 ceph-mon[25836]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:40:06.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:05 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:06.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:06 smithi114 ceph-mon[27627]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:40:06.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:06 smithi177 ceph-mon[27677]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:40:07.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:06 smithi038 ceph-mon[25836]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:40:07.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:07 smithi177 ceph-mon[27677]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:40:07.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:07 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:40:07.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:07 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:40:07.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:07 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:08.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:07 smithi038 ceph-mon[25836]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:40:08.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:07 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:40:08.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:07 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:40:08.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:07 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:08.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:07 smithi114 ceph-mon[27627]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:40:08.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:07 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:40:08.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:07 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:40:08.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:07 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:08.821 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:08 smithi177 ceph-mon[27677]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:08.821 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:08 smithi177 ceph-mon[27677]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:09.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:08 smithi038 ceph-mon[25836]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:09.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:08 smithi038 ceph-mon[25836]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:09.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:08 smithi114 ceph-mon[27627]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:09.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:08 smithi114 ceph-mon[27627]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:10.811 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:10 smithi177 ceph-mon[27677]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:10.811 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:10 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:10.811 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:10 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:10.811 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:10 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:10.811 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:10 smithi177 ceph-mon[27677]: from='client.? 172.21.15.177:0/4270716814' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "111f3ef1-97de-43aa-91a2-d7da6a2f7d1e"}]: dispatch 2024-03-26T04:40:10.811 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:10 smithi177 ceph-mon[27677]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "111f3ef1-97de-43aa-91a2-d7da6a2f7d1e"}]: dispatch 2024-03-26T04:40:10.811 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:10 smithi177 ceph-mon[27677]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "111f3ef1-97de-43aa-91a2-d7da6a2f7d1e"}]': finished 2024-03-26T04:40:10.811 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:10 smithi177 ceph-mon[27677]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:40:10.812 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:10 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:11.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:10 smithi038 ceph-mon[25836]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:11.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:10 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:11.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:10 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:11.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:10 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:11.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:10 smithi038 ceph-mon[25836]: from='client.? 172.21.15.177:0/4270716814' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "111f3ef1-97de-43aa-91a2-d7da6a2f7d1e"}]: dispatch 2024-03-26T04:40:11.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:10 smithi038 ceph-mon[25836]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "111f3ef1-97de-43aa-91a2-d7da6a2f7d1e"}]: dispatch 2024-03-26T04:40:11.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:10 smithi038 ceph-mon[25836]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "111f3ef1-97de-43aa-91a2-d7da6a2f7d1e"}]': finished 2024-03-26T04:40:11.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:10 smithi038 ceph-mon[25836]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:40:11.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:10 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:11.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:10 smithi114 ceph-mon[27627]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:11.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:10 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:11.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:10 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:11.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:10 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:11.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:10 smithi114 ceph-mon[27627]: from='client.? 172.21.15.177:0/4270716814' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "111f3ef1-97de-43aa-91a2-d7da6a2f7d1e"}]: dispatch 2024-03-26T04:40:11.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:10 smithi114 ceph-mon[27627]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "111f3ef1-97de-43aa-91a2-d7da6a2f7d1e"}]: dispatch 2024-03-26T04:40:11.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:10 smithi114 ceph-mon[27627]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "111f3ef1-97de-43aa-91a2-d7da6a2f7d1e"}]': finished 2024-03-26T04:40:11.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:10 smithi114 ceph-mon[27627]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:40:11.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:10 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:11.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:11 smithi177 ceph-mon[27677]: from='client.? 172.21.15.177:0/4163344404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:40:12.046 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:11 smithi114 ceph-mon[27627]: from='client.? 172.21.15.177:0/4163344404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:40:12.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:11 smithi038 ceph-mon[25836]: from='client.? 172.21.15.177:0/4163344404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:40:12.865 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:12 smithi114 ceph-mon[27627]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:13.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:12 smithi038 ceph-mon[25836]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:13.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:12 smithi177 ceph-mon[27677]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:14.768 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:14 smithi038 ceph-mon[25836]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:14.769 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:14.769 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:14.769 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:14 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:14.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:14 smithi177 ceph-mon[27677]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:14.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:14.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:14.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:14 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:15.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:14 smithi114 ceph-mon[27627]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:15.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:15.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:15.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:14 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:15.757 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:15 smithi038 ceph-mon[25836]: Detected new or changed devices on smithi114 2024-03-26T04:40:15.804 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:15 smithi177 ceph-mon[27677]: Detected new or changed devices on smithi114 2024-03-26T04:40:16.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:15 smithi114 ceph-mon[27627]: Detected new or changed devices on smithi114 2024-03-26T04:40:17.019 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:16 smithi177 ceph-mon[27677]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:17.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:16 smithi038 ceph-mon[25836]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:17.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:16 smithi114 ceph-mon[27627]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:17.978 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:17 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:40:17.978 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:17 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:17.994 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:17 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:40:17.995 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:17 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:18.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:17 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:40:18.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:17 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:18.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:18 smithi177 ceph-mon[27677]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:18.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:18 smithi177 ceph-mon[27677]: Deploying daemon osd.2 on smithi177 2024-03-26T04:40:18.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:18 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:18.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:18 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:18.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:18 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:18.937 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:18 smithi114 ceph-mon[27627]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:18.937 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:18 smithi114 ceph-mon[27627]: Deploying daemon osd.2 on smithi177 2024-03-26T04:40:18.937 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:18 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:18.937 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:18 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:18.937 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:18 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:19.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:18 smithi038 ceph-mon[25836]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:19.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:18 smithi038 ceph-mon[25836]: Deploying daemon osd.2 on smithi177 2024-03-26T04:40:19.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:18 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:19.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:18 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:19.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:18 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:20.344 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:20 smithi177 ceph-mon[27677]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:20.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:20 smithi038 ceph-mon[25836]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:20.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:20 smithi114 ceph-mon[27627]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:22.376 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:22 smithi177 ceph-mon[27677]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:22.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:22 smithi038 ceph-mon[25836]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:22.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:22 smithi114 ceph-mon[27627]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:24.243 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:24 smithi177 ceph-mon[27677]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:24.243 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:24 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:24.243 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:24 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:24.243 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:24 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:24.243 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:24 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:24.243 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:24 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:24.243 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:24 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:24.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:24 smithi038 ceph-mon[25836]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:24.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:24.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:24.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:24.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:24.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:24.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:24 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:24.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:24 smithi114 ceph-mon[27627]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:24.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:24 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:24.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:24 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:24.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:24 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:24.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:24 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:24.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:24 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:24.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:24 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:26.571 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:26 smithi177 ceph-mon[27677]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:26.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:26 smithi038 ceph-mon[25836]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:26.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:26 smithi114 ceph-mon[27627]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:26.888 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 2 on host 'smithi177' 2024-03-26T04:40:27.434 DEBUG:teuthology.orchestra.run.smithi177:osd.2> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T04:40:27.437 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-26T04:40:27.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T04:40:27.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:27 smithi177 ceph-mon[27677]: from='osd.2 [v2:172.21.15.177:6800/1364245230,v1:172.21.15.177:6801/1364245230]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:40:27.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:27 smithi177 ceph-mon[27677]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:40:27.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:27 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:27.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:27 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:27.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:27 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:27.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:27 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:27.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:27 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:27.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:27 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:27.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:27 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:27.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:27 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:27.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:27 smithi038 ceph-mon[25836]: from='osd.2 [v2:172.21.15.177:6800/1364245230,v1:172.21.15.177:6801/1364245230]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:40:27.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:27 smithi038 ceph-mon[25836]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:40:27.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:27 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:27.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:27 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:27.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:27 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:27.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:27 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:27.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:27 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:27.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:27 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:27.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:27 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:27.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:27 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:27.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:27 smithi114 ceph-mon[27627]: from='osd.2 [v2:172.21.15.177:6800/1364245230,v1:172.21.15.177:6801/1364245230]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:40:27.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:27 smithi114 ceph-mon[27627]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:40:27.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:27 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:27.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:27 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:27.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:27 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:27.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:27 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:27.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:27 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:27.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:27 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:27.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:27 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:27.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:27 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:27.690 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:40:28.555 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:28 smithi038 ceph-mon[25836]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:28.555 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:28 smithi038 ceph-mon[25836]: Adjusting osd_memory_target on smithi177 to 16064M 2024-03-26T04:40:28.555 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:28 smithi038 ceph-mon[25836]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:40:28.555 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:28 smithi038 ceph-mon[25836]: from='osd.2 [v2:172.21.15.177:6800/1364245230,v1:172.21.15.177:6801/1364245230]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-26T04:40:28.555 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:28 smithi038 ceph-mon[25836]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:40:28.555 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:28 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:28.555 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:28 smithi038 ceph-mon[25836]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-26T04:40:28.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:28 smithi114 ceph-mon[27627]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:28.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:28 smithi114 ceph-mon[27627]: Adjusting osd_memory_target on smithi177 to 16064M 2024-03-26T04:40:28.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:28 smithi114 ceph-mon[27627]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:40:28.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:28 smithi114 ceph-mon[27627]: from='osd.2 [v2:172.21.15.177:6800/1364245230,v1:172.21.15.177:6801/1364245230]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-26T04:40:28.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:28 smithi114 ceph-mon[27627]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:40:28.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:28 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:28.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:28 smithi114 ceph-mon[27627]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-26T04:40:28.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:28 smithi177 ceph-mon[27677]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:28.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:28 smithi177 ceph-mon[27677]: Adjusting osd_memory_target on smithi177 to 16064M 2024-03-26T04:40:28.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:28 smithi177 ceph-mon[27677]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:40:28.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:28 smithi177 ceph-mon[27677]: from='osd.2 [v2:172.21.15.177:6800/1364245230,v1:172.21.15.177:6801/1364245230]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-26T04:40:28.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:28 smithi177 ceph-mon[27677]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:40:28.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:28 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:28.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:28 smithi177 ceph-mon[27677]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-26T04:40:29.149 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:40:28 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2[34117]: 2024-03-26T04:40:28.703+0000 7f4ecb38c700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T04:40:29.149 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:40:28 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2[34117]: 2024-03-26T04:40:28.705+0000 7f4ec3177700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:40:29.549 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:40:29.549 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:29 smithi038 ceph-mon[25836]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-26T04:40:29.549 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:29 smithi038 ceph-mon[25836]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:40:29.549 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:29 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:29.549 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:29 smithi038 ceph-mon[25836]: from='osd.2 ' entity='osd.2' 2024-03-26T04:40:29.549 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:29 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:29.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:29 smithi114 ceph-mon[27627]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-26T04:40:29.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:29 smithi114 ceph-mon[27627]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:40:29.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:29 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:29.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:29 smithi114 ceph-mon[27627]: from='osd.2 ' entity='osd.2' 2024-03-26T04:40:29.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:29 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:29.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:29 smithi177 ceph-mon[27677]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-26T04:40:29.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:29 smithi177 ceph-mon[27677]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:40:29.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:29 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:29.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:29 smithi177 ceph-mon[27677]: from='osd.2 ' entity='osd.2' 2024-03-26T04:40:29.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:29 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:30.190 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"num_osds":3,"num_up_osds":2,"osd_up_since":1711428005,"num_in_osds":3,"osd_in_since":1711428010,"num_remapped_pgs":0} 2024-03-26T04:40:30.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:30 smithi038 ceph-mon[25836]: purged_snaps scrub starts 2024-03-26T04:40:30.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:30 smithi038 ceph-mon[25836]: purged_snaps scrub ok 2024-03-26T04:40:30.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:30 smithi038 ceph-mon[25836]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:30.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:30 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2666893164' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:40:30.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:30 smithi038 ceph-mon[25836]: osd.2 [v2:172.21.15.177:6800/1364245230,v1:172.21.15.177:6801/1364245230] boot 2024-03-26T04:40:30.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:30 smithi038 ceph-mon[25836]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:40:30.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:30 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:30.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:30 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:30.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:30 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:30.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:30 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:30 smithi114 ceph-mon[27627]: purged_snaps scrub starts 2024-03-26T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:30 smithi114 ceph-mon[27627]: purged_snaps scrub ok 2024-03-26T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:30 smithi114 ceph-mon[27627]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:30 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2666893164' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:30 smithi114 ceph-mon[27627]: osd.2 [v2:172.21.15.177:6800/1364245230,v1:172.21.15.177:6801/1364245230] boot 2024-03-26T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:30 smithi114 ceph-mon[27627]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:30 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:30 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:30.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:30 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:30.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:30 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:30.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:30 smithi177 ceph-mon[27677]: purged_snaps scrub starts 2024-03-26T04:40:30.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:30 smithi177 ceph-mon[27677]: purged_snaps scrub ok 2024-03-26T04:40:30.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:30 smithi177 ceph-mon[27677]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:40:30.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:30 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2666893164' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:40:30.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:30 smithi177 ceph-mon[27677]: osd.2 [v2:172.21.15.177:6800/1364245230,v1:172.21.15.177:6801/1364245230] boot 2024-03-26T04:40:30.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:30 smithi177 ceph-mon[27677]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:40:30.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:30 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:30.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:30 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:30.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:30 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:30.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:30 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:31.191 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T04:40:31.435 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:40:31.557 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:31 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:40:31.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:31 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:40:31.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:31 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:40:32.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:32 smithi038 ceph-mon[25836]: pgmap v84: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:32.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:32 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:40:32.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:32 smithi038 ceph-mon[25836]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:40:32.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:32 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:40:32.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:32 smithi114 ceph-mon[27627]: pgmap v84: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:32.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:32 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:40:32.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:32 smithi114 ceph-mon[27627]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:40:32.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:32 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:40:32.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:32 smithi177 ceph-mon[27677]: pgmap v84: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:32.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:32 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:40:32.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:32 smithi177 ceph-mon[27677]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:40:32.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:32 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:40:32.888 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:40:32 smithi038 sudo[40365]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-26T04:40:32.888 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:40:32 smithi038 sudo[40365]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:40:32.888 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:40:32 smithi038 sudo[40365]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:40:32.888 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:40:32 smithi038 sudo[40365]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:40:33.238 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:40:32 smithi038 sudo[40386]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-26T04:40:33.239 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:40:32 smithi038 sudo[40386]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:40:33.239 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:40:32 smithi038 sudo[40386]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:40:33.239 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:40:32 smithi038 sudo[40386]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:40:33.433 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:33 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:40:33.434 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:33 smithi114 ceph-mon[27627]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:40:33.434 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:33 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:33.434 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:40:33 smithi114 sudo[34393]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-26T04:40:33.434 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:40:33 smithi114 sudo[34393]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:40:33.434 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:40:33 smithi114 sudo[34393]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:40:33.434 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:40:33 smithi114 sudo[34393]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:40:33.543 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:33 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:40:33.543 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:33 smithi038 ceph-mon[25836]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:40:33.543 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:33 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:33.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:33 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:40:33.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:33 smithi177 ceph-mon[27677]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:40:33.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:33 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:33.853 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:40:33.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:40:33 smithi114 sudo[34396]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-26T04:40:33.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:40:33 smithi114 sudo[34396]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:40:33.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:40:33 smithi114 sudo[34396]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:40:33.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:40:33 smithi114 sudo[34396]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:40:33.953 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:40:33 smithi177 sudo[35110]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-26T04:40:33.953 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:40:33 smithi177 sudo[35110]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:40:33.953 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:40:33 smithi177 sudo[35110]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:40:33.953 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:40:33 smithi177 sudo[35110]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:40:34.399 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:40:33 smithi177 sudo[35113]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-26T04:40:34.399 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:40:33 smithi177 sudo[35113]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:40:34.399 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:40:33 smithi177 sudo[35113]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:40:34.399 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:40:33 smithi177 sudo[35113]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:40:34.423 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 sudo[40508]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:40:34.424 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 sudo[40508]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:40:34.424 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 sudo[40508]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:40:34.424 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 sudo[40508]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:40:34.448 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":21,"num_osds":3,"num_up_osds":3,"osd_up_since":1711428029,"num_in_osds":3,"osd_in_since":1711428010,"num_remapped_pgs":0} 2024-03-26T04:40:34.449 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:40:34.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:34 smithi177 ceph-mon[27677]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:34.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:34 smithi177 ceph-mon[27677]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:40:34.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:34 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:34.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:34 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2793264611' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:40:34.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:34 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:34.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:34 smithi177 ceph-mon[27677]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:40:34.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 sudo[40530]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:40:34.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 sudo[40530]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:40:34.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 sudo[40530]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:40:34.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 sudo[40530]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:40:34.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 ceph-mon[25836]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:34.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 ceph-mon[25836]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:40:34.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:34.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2793264611' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:40:34.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:34.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:34 smithi038 ceph-mon[25836]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:40:34.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:34 smithi114 ceph-mon[27627]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:34.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:34 smithi114 ceph-mon[27627]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:40:34.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:34 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:34.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:34 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2793264611' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:40:34.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:34 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:34.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:34 smithi114 ceph-mon[27627]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:40:35.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:34 smithi177 sudo[35116]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:40:35.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:34 smithi177 sudo[35116]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:40:35.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:34 smithi177 sudo[35116]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:40:35.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:34 smithi177 sudo[35116]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:40:35.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 sudo[35119]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:40:35.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 sudo[35119]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:40:35.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 sudo[35119]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:40:35.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 sudo[35119]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:40:35.472 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:40:35.559 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 sudo[34467]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:40:35.559 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 sudo[34467]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:40:35.559 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 sudo[34467]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:40:35.559 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 sudo[34467]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:40:35.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:40:35.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:35.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:40:35.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:40:35.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:35 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:40:35.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 sudo[34470]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:40:35.921 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 sudo[34470]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 sudo[34470]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 sudo[34470]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:35 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:35.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:35.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:40:35.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:35 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:40:36.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:36 smithi038 ceph-mon[25836]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:36.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:36 smithi038 ceph-mon[25836]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:40:36.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:36 smithi038 ceph-mon[25836]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:40:36.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:36 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:36.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:36 smithi114 ceph-mon[27627]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:36.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:36 smithi114 ceph-mon[27627]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:40:36.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:36 smithi114 ceph-mon[27627]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:40:36.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:36 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:36.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:36 smithi177 ceph-mon[27677]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:36.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:36 smithi177 ceph-mon[27677]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:40:36.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:36 smithi177 ceph-mon[27677]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:40:36.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:36 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:38.000 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:40:38.000 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":21,"fsid":"56caa506-eb2a-11ee-95d0-87774f69a715","created":"2024-03-26T04:36:55.633203+0000","modified":"2024-03-26T04:40:33.287385+0000","last_up_change":"2024-03-26T04:40:29.706706+0000","last_in_change":"2024-03-26T04:40:10.202973+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:40:30.996140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"36825eb5-ff6e-4e19-adab-791266b00c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":755874714},{"type":"v1","addr":"172.21.15.38:6803","nonce":755874714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":755874714},{"type":"v1","addr":"172.21.15.38:6805","nonce":755874714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":755874714},{"type":"v1","addr":"172.21.15.38:6809","nonce":755874714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":755874714},{"type":"v1","addr":"172.21.15.38:6807","nonce":755874714}]},"public_addr":"172.21.15.38:6803/755874714","cluster_addr":"172.21.15.38:6805/755874714","heartbeat_back_addr":"172.21.15.38:6809/755874714","heartbeat_front_addr":"172.21.15.38:6807/755874714","state":["exists","up"]},{"osd":1,"uuid":"631b8290-ee85-4525-be34-13e99b5e9573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3929328856},{"type":"v1","addr":"172.21.15.114:6801","nonce":3929328856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3929328856},{"type":"v1","addr":"172.21.15.114:6803","nonce":3929328856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3929328856},{"type":"v1","addr":"172.21.15.114:6807","nonce":3929328856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3929328856},{"type":"v1","addr":"172.21.15.114:6805","nonce":3929328856}]},"public_addr":"172.21.15.114:6801/3929328856","cluster_addr":"172.21.15.114:6803/3929328856","heartbeat_back_addr":"172.21.15.114:6807/3929328856","heartbeat_front_addr":"172.21.15.114:6805/3929328856","state":["exists","up"]},{"osd":2,"uuid":"111f3ef1-97de-43aa-91a2-d7da6a2f7d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1364245230},{"type":"v1","addr":"172.21.15.177:6801","nonce":1364245230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1364245230},{"type":"v1","addr":"172.21.15.177:6803","nonce":1364245230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1364245230},{"type":"v1","addr":"172.21.15.177:6807","nonce":1364245230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1364245230},{"type":"v1","addr":"172.21.15.177:6805","nonce":1364245230}]},"public_addr":"172.21.15.177:6801/1364245230","cluster_addr":"172.21.15.177:6803/1364245230","heartbeat_back_addr":"172.21.15.177:6807/1364245230","heartbeat_front_addr":"172.21.15.177:6805/1364245230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:39:38.055997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:40:02.656316+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:40:27.326253+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/1018230916":"2024-03-27T04:37:58.646615+0000","172.21.15.38:6801/3938089582":"2024-03-27T04:37:58.646615+0000","172.21.15.38:0/4113838286":"2024-03-27T04:37:58.646615+0000","172.21.15.38:0/4140847":"2024-03-27T04:37:30.956666+0000","172.21.15.38:0/1308956235":"2024-03-27T04:37:30.956666+0000","172.21.15.38:0/378116029":"2024-03-27T04:37:30.956666+0000","172.21.15.38:6800/3938089582":"2024-03-27T04:37:58.646615+0000","172.21.15.38:6801/2491069991":"2024-03-27T04:37:30.956666+0000","172.21.15.38:6800/2491069991":"2024-03-27T04:37:30.956666+0000","172.21.15.38:0/329205786":"2024-03-27T04:37:58.646615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:40:38.609 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T04:40:30.996140+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T04:40:38.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T04:40:38.719 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:38 smithi114 ceph-mon[27627]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:38.719 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:38.719 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:38.719 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:38 smithi114 ceph-mon[27627]: Detected new or changed devices on smithi177 2024-03-26T04:40:38.719 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:38 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:38.719 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:38 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/28931774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:40:38.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:38 smithi038 ceph-mon[25836]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:38.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:38.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:38.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:38 smithi038 ceph-mon[25836]: Detected new or changed devices on smithi177 2024-03-26T04:40:38.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:38 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:38.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:38 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/28931774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:40:38.847 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:40:38.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:38 smithi177 ceph-mon[27677]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:38.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:38.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:38.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:38 smithi177 ceph-mon[27677]: Detected new or changed devices on smithi177 2024-03-26T04:40:38.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:38 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:38.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:38 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/28931774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:40:40.697 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-03-26T04:40:40.697 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:40 smithi038 ceph-mon[25836]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:40.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:40 smithi114 ceph-mon[27627]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:40.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:40 smithi177 ceph-mon[27677]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:41.333 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi038 2024-03-26T04:40:41.334 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi038=foo.a' 2024-03-26T04:40:41.890 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:41 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/1658365948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:40:41.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:41 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/1658365948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:40:41.946 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:40:42.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:41 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/1658365948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:40:42.863 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:42 smithi177 ceph-mon[27677]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:42.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:42 smithi114 ceph-mon[27627]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:43.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:42 smithi038 ceph-mon[25836]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:44.360 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled rgw.foo.a update... 2024-03-26T04:40:44.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:44 smithi177 ceph-mon[27677]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:44.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:44 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:44.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:44 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:44.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:44 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:44.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:44 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:44.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:44 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:44.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:44 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:40:44.710 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:44 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:40:44.710 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:44 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:44.710 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:44 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:44.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:44 smithi114 ceph-mon[27627]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:44.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:44 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:44.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:44 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:44.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:44 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:44.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:44 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:44.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:44 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:44.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:44 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:40:44.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:44 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:40:44.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:44 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:44.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:44 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:45.012 DEBUG:teuthology.orchestra.run.smithi038:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@rgw.foo.a.service 2024-03-26T04:40:45.014 INFO:tasks.cephadm:Adding prometheus.a on smithi114 2024-03-26T04:40:45.015 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi114=a' 2024-03-26T04:40:45.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:44 smithi038 ceph-mon[25836]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:45.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:44 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:45.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:44 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:45.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:44 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:45.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:44 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:45.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:44 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:45.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:44 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:40:45.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:44 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:40:45.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:44 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:45.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:44 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:45.250 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:40:45.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:45 smithi177 ceph-mon[27677]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi038=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:45.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:45 smithi177 ceph-mon[27677]: Saving service rgw.foo.a spec with placement smithi038=foo.a;count:1 2024-03-26T04:40:45.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:45 smithi177 ceph-mon[27677]: Deploying daemon rgw.foo.a on smithi038 2024-03-26T04:40:45.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:45 smithi038 ceph-mon[25836]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi038=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:45.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:45 smithi038 ceph-mon[25836]: Saving service rgw.foo.a spec with placement smithi038=foo.a;count:1 2024-03-26T04:40:45.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:45 smithi038 ceph-mon[25836]: Deploying daemon rgw.foo.a on smithi038 2024-03-26T04:40:45.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:45 smithi114 ceph-mon[27627]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi038=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:45.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:45 smithi114 ceph-mon[27627]: Saving service rgw.foo.a spec with placement smithi038=foo.a;count:1 2024-03-26T04:40:45.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:45 smithi114 ceph-mon[27627]: Deploying daemon rgw.foo.a on smithi038 2024-03-26T04:40:46.756 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:46 smithi038 ceph-mon[25836]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:46.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:46 smithi114 ceph-mon[27627]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:46.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:46 smithi177 ceph-mon[27677]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:46.958 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled prometheus update... 2024-03-26T04:40:47.065 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 26 04:40:46 smithi038 podman[41368]: 2024-03-26 04:40:46.987963503 +0000 UTC m=+0.346606408 container start d6b074fa023ed5bde8fa6921494e80af98efc6ef1ee3ccd4b20994f60d95dc2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:40:47.065 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 26 04:40:47 smithi038 bash[41368]: d6b074fa023ed5bde8fa6921494e80af98efc6ef1ee3ccd4b20994f60d95dc2b 2024-03-26T04:40:47.330 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 26 04:40:47 smithi038 systemd[1]: Started Ceph rgw.foo.a for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:40:47.531 DEBUG:teuthology.orchestra.run.smithi114:prometheus.a> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T04:40:47.534 INFO:tasks.cephadm:Adding node-exporter.a on smithi038 2024-03-26T04:40:47.534 INFO:tasks.cephadm:Adding node-exporter.b on smithi114 2024-03-26T04:40:47.534 INFO:tasks.cephadm:Adding node-exporter.c on smithi177 2024-03-26T04:40:47.534 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '3;smithi038=a;smithi114=b;smithi177=c' 2024-03-26T04:40:47.768 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:40:48.145 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:48.145 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:48.145 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: Saving service prometheus spec with placement smithi114=a;count:1 2024-03-26T04:40:48.145 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.145 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.145 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.145 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.145 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: Saving service rgw.foo.a spec with placement smithi038=foo.a;count:1 2024-03-26T04:40:48.145 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:48.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:48.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:48.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:47 smithi177 ceph-mon[27677]: Checking dashboard <-> RGW credentials 2024-03-26T04:40:48.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:48.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:48.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: Saving service prometheus spec with placement smithi114=a;count:1 2024-03-26T04:40:48.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: Saving service rgw.foo.a spec with placement smithi038=foo.a;count:1 2024-03-26T04:40:48.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:48.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:48.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:48.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:47 smithi038 ceph-mon[25836]: Checking dashboard <-> RGW credentials 2024-03-26T04:40:48.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:48.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:48.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: Saving service prometheus spec with placement smithi114=a;count:1 2024-03-26T04:40:48.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: Saving service rgw.foo.a spec with placement smithi038=foo.a;count:1 2024-03-26T04:40:48.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:48.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:48.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:48.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:48.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:47 smithi114 ceph-mon[27627]: Checking dashboard <-> RGW credentials 2024-03-26T04:40:49.137 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:48 smithi177 ceph-mon[27677]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T04:40:49.137 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:48 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2604472952' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:40:49.137 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:48 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/3596908443' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:40:49.137 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:48 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:49.311 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:48 smithi038 ceph-mon[25836]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T04:40:49.311 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:48 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2604472952' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:40:49.311 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:48 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/3596908443' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:40:49.311 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:48 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:49.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:48 smithi114 ceph-mon[27627]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T04:40:49.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:48 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2604472952' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:40:49.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:48 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/3596908443' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:40:49.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:48 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:49.610 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled node-exporter update... 2024-03-26T04:40:50.159 DEBUG:teuthology.orchestra.run.smithi038:node-exporter.a> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T04:40:50.161 DEBUG:teuthology.orchestra.run.smithi114:node-exporter.b> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T04:40:50.164 DEBUG:teuthology.orchestra.run.smithi177:node-exporter.c> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@node-exporter.c.service 2024-03-26T04:40:50.166 INFO:tasks.cephadm:Adding alertmanager.a on smithi038 2024-03-26T04:40:50.166 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi038=a' 2024-03-26T04:40:50.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:49 smithi038 ceph-mon[25836]: pgmap v97: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:50.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:49 smithi038 ceph-mon[25836]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:40:50.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:49 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2604472952' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:40:50.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:49 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/3596908443' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:40:50.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:49 smithi038 ceph-mon[25836]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T04:40:50.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:49 smithi038 ceph-mon[25836]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi038=a;smithi114=b;smithi177=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:50.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:49 smithi038 ceph-mon[25836]: Saving service node-exporter spec with placement smithi038=a;smithi114=b;smithi177=c;count:3 2024-03-26T04:40:50.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:49 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:50.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:49 smithi177 ceph-mon[27677]: pgmap v97: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:50.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:49 smithi177 ceph-mon[27677]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:40:50.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:49 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2604472952' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:40:50.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:49 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/3596908443' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:40:50.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:49 smithi177 ceph-mon[27677]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T04:40:50.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:49 smithi177 ceph-mon[27677]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi038=a;smithi114=b;smithi177=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:50.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:49 smithi177 ceph-mon[27677]: Saving service node-exporter spec with placement smithi038=a;smithi114=b;smithi177=c;count:3 2024-03-26T04:40:50.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:49 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:50.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:49 smithi114 ceph-mon[27627]: pgmap v97: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:50.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:49 smithi114 ceph-mon[27627]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:40:50.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:49 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2604472952' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:40:50.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:49 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/3596908443' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:40:50.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:49 smithi114 ceph-mon[27627]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T04:40:50.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:49 smithi114 ceph-mon[27627]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi038=a;smithi114=b;smithi177=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:50.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:49 smithi114 ceph-mon[27627]: Saving service node-exporter spec with placement smithi038=a;smithi114=b;smithi177=c;count:3 2024-03-26T04:40:50.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:49 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:50.416 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:40:51.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:51 smithi038 ceph-mon[25836]: osdmap e24: 3 total, 3 up, 3 in 2024-03-26T04:40:51.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:51 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:40:51.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:51 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2583047573' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:40:51.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:51 smithi038 ceph-mon[25836]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:40:51.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:51 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:51.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:51 smithi114 ceph-mon[27627]: osdmap e24: 3 total, 3 up, 3 in 2024-03-26T04:40:51.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:51 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:40:51.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:51 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2583047573' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:40:51.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:51 smithi114 ceph-mon[27627]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:40:51.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:51 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:51.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:51 smithi177 ceph-mon[27677]: osdmap e24: 3 total, 3 up, 3 in 2024-03-26T04:40:51.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:51 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:40:51.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:51 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2583047573' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:40:51.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:51 smithi177 ceph-mon[27677]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:40:51.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:51 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:52.158 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled alertmanager update... 2024-03-26T04:40:52.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:52 smithi038 ceph-mon[25836]: pgmap v100: 65 pgs: 32 creating+peering, 33 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:40:52.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:52 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:40:52.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:52 smithi038 ceph-mon[25836]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:40:52.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:52 smithi038 ceph-mon[25836]: osdmap e25: 3 total, 3 up, 3 in 2024-03-26T04:40:52.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:52 smithi114 ceph-mon[27627]: pgmap v100: 65 pgs: 32 creating+peering, 33 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:40:52.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:52 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:40:52.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:52 smithi114 ceph-mon[27627]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:40:52.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:52 smithi114 ceph-mon[27627]: osdmap e25: 3 total, 3 up, 3 in 2024-03-26T04:40:52.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:52 smithi177 ceph-mon[27677]: pgmap v100: 65 pgs: 32 creating+peering, 33 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:40:52.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:52 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:40:52.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:52 smithi177 ceph-mon[27677]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:40:52.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:52 smithi177 ceph-mon[27677]: osdmap e25: 3 total, 3 up, 3 in 2024-03-26T04:40:52.746 DEBUG:teuthology.orchestra.run.smithi038:alertmanager.a> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T04:40:52.749 INFO:tasks.cephadm:Adding grafana.a on smithi114 2024-03-26T04:40:52.749 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi114=a' 2024-03-26T04:40:52.983 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:40:53.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: osdmap e26: 3 total, 3 up, 3 in 2024-03-26T04:40:53.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2583047573' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:40:53.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:40:53.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:40:53.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi038=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:53.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: Saving service alertmanager spec with placement smithi038=a;count:1 2024-03-26T04:40:53.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:40:53.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:40:53.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:53 smithi038 ceph-mon[25836]: osdmap e27: 3 total, 3 up, 3 in 2024-03-26T04:40:53.338 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: osdmap e26: 3 total, 3 up, 3 in 2024-03-26T04:40:53.338 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2583047573' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:40:53.338 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:40:53.338 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:40:53.338 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi038=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:53.338 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: Saving service alertmanager spec with placement smithi038=a;count:1 2024-03-26T04:40:53.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:40:53.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:40:53.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:53 smithi177 ceph-mon[27677]: osdmap e27: 3 total, 3 up, 3 in 2024-03-26T04:40:53.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: osdmap e26: 3 total, 3 up, 3 in 2024-03-26T04:40:53.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2583047573' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:40:53.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:40:53.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:40:53.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi038=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:53.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: Saving service alertmanager spec with placement smithi038=a;count:1 2024-03-26T04:40:53.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:53.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:40:53.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:40:53.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:53 smithi114 ceph-mon[27627]: osdmap e27: 3 total, 3 up, 3 in 2024-03-26T04:40:54.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:54 smithi177 ceph-mon[27677]: pgmap v103: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T04:40:54.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:54 smithi177 ceph-mon[27677]: osdmap e28: 3 total, 3 up, 3 in 2024-03-26T04:40:54.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:54 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:40:54.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:54 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2583047573' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:40:54.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:54 smithi177 ceph-mon[27677]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:40:54.782 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled grafana update... 2024-03-26T04:40:54.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:54 smithi038 ceph-mon[25836]: pgmap v103: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T04:40:54.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:54 smithi038 ceph-mon[25836]: osdmap e28: 3 total, 3 up, 3 in 2024-03-26T04:40:54.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:54 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:40:54.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:54 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2583047573' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:40:54.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:54 smithi038 ceph-mon[25836]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:40:54.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:54 smithi114 ceph-mon[27627]: pgmap v103: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T04:40:54.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:54 smithi114 ceph-mon[27627]: osdmap e28: 3 total, 3 up, 3 in 2024-03-26T04:40:54.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:54 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:40:54.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:54 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2583047573' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:40:54.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:54 smithi114 ceph-mon[27627]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:40:55.349 DEBUG:teuthology.orchestra.run.smithi114:grafana.a> sudo journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T04:40:55.352 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T04:40:55.352 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T04:40:55.603 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:40:56.020 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:55 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.020 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:55 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.020 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:55 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.020 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:55 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.020 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:55 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:40:56.020 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:55 smithi038 ceph-mon[25836]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:40:56.021 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:55 smithi038 ceph-mon[25836]: osdmap e29: 3 total, 3 up, 3 in 2024-03-26T04:40:56.021 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:55 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2583047573' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:40:56.021 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:55 smithi038 ceph-mon[25836]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:40:56.021 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:55 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:40:56.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:55 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:55 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:55 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:55 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:55 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:40:56.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:55 smithi114 ceph-mon[27627]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:40:56.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:55 smithi114 ceph-mon[27627]: osdmap e29: 3 total, 3 up, 3 in 2024-03-26T04:40:56.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:55 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2583047573' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:40:56.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:55 smithi114 ceph-mon[27627]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:40:56.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:55 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:40:56.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:55 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:55 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:55 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:55 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:55 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:40:56.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:55 smithi177 ceph-mon[27677]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:40:56.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:55 smithi177 ceph-mon[27677]: osdmap e29: 3 total, 3 up, 3 in 2024-03-26T04:40:56.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:55 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2583047573' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:40:56.150 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:55 smithi177 ceph-mon[27677]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:40:56.150 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:55 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:40:56.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:56 smithi038 ceph-mon[25836]: pgmap v106: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:56.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:56 smithi038 ceph-mon[25836]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:56.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:56 smithi038 ceph-mon[25836]: Saving service grafana spec with placement smithi114=a;count:1 2024-03-26T04:40:56.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:56 smithi038 ceph-mon[25836]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:40:56.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:56 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:40:56.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:56 smithi038 ceph-mon[25836]: osdmap e30: 3 total, 3 up, 3 in 2024-03-26T04:40:56.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:56 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:56 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:56 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.958 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:56 smithi114 ceph-mon[27627]: pgmap v106: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:56.959 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:56 smithi114 ceph-mon[27627]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:56.959 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:56 smithi114 ceph-mon[27627]: Saving service grafana spec with placement smithi114=a;count:1 2024-03-26T04:40:56.959 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:56 smithi114 ceph-mon[27627]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:40:56.959 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:56 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:40:56.959 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:56 smithi114 ceph-mon[27627]: osdmap e30: 3 total, 3 up, 3 in 2024-03-26T04:40:56.959 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:56 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.959 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:56 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:56.959 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:56 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:57.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:56 smithi177 ceph-mon[27677]: pgmap v106: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:40:57.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:56 smithi177 ceph-mon[27677]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:57.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:56 smithi177 ceph-mon[27677]: Saving service grafana spec with placement smithi114=a;count:1 2024-03-26T04:40:57.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:56 smithi177 ceph-mon[27677]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:40:57.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:56 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2191763134' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:40:57.150 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:56 smithi177 ceph-mon[27677]: osdmap e30: 3 total, 3 up, 3 in 2024-03-26T04:40:57.150 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:56 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:57.150 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:56 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:57.150 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:56 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:57.580 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2024-03-26T04:40:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDZUQJmNot+IhAAUhrI5c9PnU96JX3h7yNe3g== 2024-03-26T04:40:57.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:57 smithi038 ceph-mon[25836]: Deploying daemon prometheus.a on smithi114 2024-03-26T04:40:57.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:57 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/3278251765' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:40:57.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:57 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/3278251765' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:40:57.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:57 smithi177 ceph-mon[27677]: Deploying daemon prometheus.a on smithi114 2024-03-26T04:40:57.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:57 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/3278251765' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:40:57.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:57 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/3278251765' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:40:57.969 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:57 smithi114 ceph-mon[27627]: Deploying daemon prometheus.a on smithi114 2024-03-26T04:40:57.969 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:57 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/3278251765' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:40:57.969 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:57 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/3278251765' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:40:58.262 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:40:58.262 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T04:40:58.262 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T04:40:58.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T04:40:58.543 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:40:58.927 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:58 smithi114 ceph-mon[27627]: pgmap v109: 129 pgs: 129 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:40:58.927 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:58 smithi114 ceph-mon[27627]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:40:58.927 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:58 smithi114 ceph-mon[27627]: Cluster is now healthy 2024-03-26T04:40:58.928 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:58 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:58.928 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:58 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:58.928 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:40:58 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:59.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:58 smithi038 ceph-mon[25836]: pgmap v109: 129 pgs: 129 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:40:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:58 smithi038 ceph-mon[25836]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:40:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:58 smithi038 ceph-mon[25836]: Cluster is now healthy 2024-03-26T04:40:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:58 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:58 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:40:58 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:59.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:58 smithi177 ceph-mon[27677]: pgmap v109: 129 pgs: 129 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:40:59.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:58 smithi177 ceph-mon[27677]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:40:59.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:58 smithi177 ceph-mon[27677]: Cluster is now healthy 2024-03-26T04:40:59.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:58 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:59.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:58 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:40:59.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:40:58 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:41:00.795 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:00 smithi114 ceph-mon[27627]: pgmap v110: 129 pgs: 129 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 360 B/s wr, 2 op/s 2024-03-26T04:41:00.823 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:00 smithi177 ceph-mon[27677]: pgmap v110: 129 pgs: 129 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 360 B/s wr, 2 op/s 2024-03-26T04:41:01.065 INFO:teuthology.orchestra.run.smithi114.stdout:[client.1] 2024-03-26T04:41:01.065 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDdUQJmo3OaAxAAaPIinx9SHl/XBS5xISHbvA== 2024-03-26T04:41:01.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:00 smithi038 ceph-mon[25836]: pgmap v110: 129 pgs: 129 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 360 B/s wr, 2 op/s 2024-03-26T04:41:01.869 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:01 smithi114 ceph-mon[27627]: from='client.? 172.21.15.114:0/1720007840' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:41:01.869 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:01 smithi114 ceph-mon[27627]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:41:01.869 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:01 smithi114 ceph-mon[27627]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:41:01.896 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-26T04:41:01.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T04:41:01.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T04:41:01.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:01 smithi177 ceph-mon[27677]: from='client.? 172.21.15.114:0/1720007840' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:41:01.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:01 smithi177 ceph-mon[27677]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:41:01.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:01 smithi177 ceph-mon[27677]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:41:01.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T04:41:02.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:01 smithi038 ceph-mon[25836]: from='client.? 172.21.15.114:0/1720007840' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:41:02.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:01 smithi038 ceph-mon[25836]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:41:02.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:01 smithi038 ceph-mon[25836]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:41:02.183 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:41:03.043 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:02 smithi177 ceph-mon[27677]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 108 KiB/s rd, 5.4 KiB/s wr, 241 op/s 2024-03-26T04:41:03.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:02 smithi038 ceph-mon[25836]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 108 KiB/s rd, 5.4 KiB/s wr, 241 op/s 2024-03-26T04:41:03.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:02 smithi114 ceph-mon[27627]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 108 KiB/s rd, 5.4 KiB/s wr, 241 op/s 2024-03-26T04:41:04.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:03 smithi038 ceph-mon[25836]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 90 KiB/s rd, 4.5 KiB/s wr, 200 op/s 2024-03-26T04:41:04.089 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:03 smithi177 ceph-mon[27677]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 90 KiB/s rd, 4.5 KiB/s wr, 200 op/s 2024-03-26T04:41:04.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:03 smithi114 ceph-mon[27627]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 90 KiB/s rd, 4.5 KiB/s wr, 200 op/s 2024-03-26T04:41:04.694 INFO:teuthology.orchestra.run.smithi177.stdout:[client.2] 2024-03-26T04:41:04.694 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQDgUQJmqJUjKRAAXWU8jSiZp3fK8m38dFetDQ== 2024-03-26T04:41:04.994 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:04 smithi177 ceph-mon[27677]: from='client.? 172.21.15.177:0/501133246' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:41:04.994 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:04 smithi177 ceph-mon[27677]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:41:04.994 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:04 smithi177 ceph-mon[27677]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:41:05.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:04 smithi038 ceph-mon[25836]: from='client.? 172.21.15.177:0/501133246' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:41:05.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:04 smithi038 ceph-mon[25836]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:41:05.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:04 smithi038 ceph-mon[25836]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:41:05.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:04 smithi114 ceph-mon[27627]: from='client.? 172.21.15.177:0/501133246' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:41:05.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:04 smithi114 ceph-mon[27627]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:41:05.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:04 smithi114 ceph-mon[27627]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:41:05.246 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-26T04:41:05.246 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-26T04:41:05.246 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-26T04:41:05.287 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T04:41:05.288 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T04:41:05.288 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T04:41:05.538 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:05.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:05 smithi038 ceph-mon[25836]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 75 KiB/s rd, 3.7 KiB/s wr, 166 op/s 2024-03-26T04:41:06.061 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:05 smithi114 ceph-mon[27627]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 75 KiB/s rd, 3.7 KiB/s wr, 166 op/s 2024-03-26T04:41:06.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:05 smithi177 ceph-mon[27677]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 75 KiB/s rd, 3.7 KiB/s wr, 166 op/s 2024-03-26T04:41:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:41:07.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:07 smithi038 ceph-mon[25836]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 67 KiB/s rd, 3.2 KiB/s wr, 149 op/s 2024-03-26T04:41:07.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:07 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/238070500' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:41:07.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:07 smithi114 ceph-mon[27627]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 67 KiB/s rd, 3.2 KiB/s wr, 149 op/s 2024-03-26T04:41:07.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:07 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/238070500' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:41:08.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:07 smithi177 ceph-mon[27677]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 67 KiB/s rd, 3.2 KiB/s wr, 149 op/s 2024-03-26T04:41:08.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:07 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/238070500' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:41:08.223 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":63737987},{"type":"v1","addr":"172.21.15.38:6801","nonce":63737987}]},"active_addr":"172.21.15.38:6801/63737987","active_change":"2024-03-26T04:37:58.646771+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.38:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":1578911713}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":716812923}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":3569585296}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":1143158148}]}]} 2024-03-26T04:41:08.229 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T04:41:08.229 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T04:41:08.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:41:08.469 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:08.896 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:08 smithi114 systemd[1]: Starting Ceph prometheus.a for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:41:09.208 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:08 smithi114 podman[35594]: 2024-03-26 04:41:08.95229324 +0000 UTC m=+0.113395586 container create bcee2c3113c0455d43a7500d7be27ec435accd786f11e42c42ec15ddc4f0b8ed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:09.208 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:08 smithi114 podman[35594]: 2024-03-26 04:41:08.856927159 +0000 UTC m=+0.018029507 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T04:41:09.208 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 podman[35594]: 2024-03-26 04:41:09.179308104 +0000 UTC m=+0.340410453 container init bcee2c3113c0455d43a7500d7be27ec435accd786f11e42c42ec15ddc4f0b8ed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:09.208 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 podman[35594]: 2024-03-26 04:41:09.184944657 +0000 UTC m=+0.346047004 container start bcee2c3113c0455d43a7500d7be27ec435accd786f11e42c42ec15ddc4f0b8ed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:09.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.208Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T04:41:09.495 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.208Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T04:41:09.495 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.208Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi114 (none))" 2024-03-26T04:41:09.495 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.208Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T04:41:09.495 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.208Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T04:41:09.495 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.210Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T04:41:09.495 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.210Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T04:41:09.495 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.211Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T04:41:09.495 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.211Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T04:41:09.495 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.212Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T04:41:09.495 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.212Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.421µs 2024-03-26T04:41:09.495 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.212Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T04:41:09.495 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.213Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T04:41:09.496 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.213Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=24.535µs wal_replay_duration=704.593µs wbl_replay_duration=140ns total_replay_duration=749.563µs 2024-03-26T04:41:09.496 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.214Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T04:41:09.496 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.214Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T04:41:09.496 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.214Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T04:41:09.496 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.231Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.619871ms db_storage=1.057µs remote_storage=1.042µs web_handler=400ns query_engine=477ns scrape=661.366µs scrape_sd=66.259µs notify=880ns notify_sd=1.298µs rules=15.720474ms tracing=6.652µs 2024-03-26T04:41:09.496 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.231Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T04:41:09.496 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:41:09.231Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T04:41:09.496 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 bash[35594]: bcee2c3113c0455d43a7500d7be27ec435accd786f11e42c42ec15ddc4f0b8ed 2024-03-26T04:41:09.496 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:41:09 smithi114 systemd[1]: Started Ceph prometheus.a for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:41:10.394 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:41:10.394 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":30,"fsid":"56caa506-eb2a-11ee-95d0-87774f69a715","created":"2024-03-26T04:36:55.633203+0000","modified":"2024-03-26T04:40:56.017556+0000","last_up_change":"2024-03-26T04:40:29.706706+0000","last_in_change":"2024-03-26T04:40:10.202973+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:40:30.996140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T04:40:47.274735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-26T04:40:49.303156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-26T04:40:51.264415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-26T04:40:53.320116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"36825eb5-ff6e-4e19-adab-791266b00c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":755874714},{"type":"v1","addr":"172.21.15.38:6803","nonce":755874714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":755874714},{"type":"v1","addr":"172.21.15.38:6805","nonce":755874714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":755874714},{"type":"v1","addr":"172.21.15.38:6809","nonce":755874714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":755874714},{"type":"v1","addr":"172.21.15.38:6807","nonce":755874714}]},"public_addr":"172.21.15.38:6803/755874714","cluster_addr":"172.21.15.38:6805/755874714","heartbeat_back_addr":"172.21.15.38:6809/755874714","heartbeat_front_addr":"172.21.15.38:6807/755874714","state":["exists","up"]},{"osd":1,"uuid":"631b8290-ee85-4525-be34-13e99b5e9573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3929328856},{"type":"v1","addr":"172.21.15.114:6801","nonce":3929328856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3929328856},{"type":"v1","addr":"172.21.15.114:6803","nonce":3929328856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3929328856},{"type":"v1","addr":"172.21.15.114:6807","nonce":3929328856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3929328856},{"type":"v1","addr":"172.21.15.114:6805","nonce":3929328856}]},"public_addr":"172.21.15.114:6801/3929328856","cluster_addr":"172.21.15.114:6803/3929328856","heartbeat_back_addr":"172.21.15.114:6807/3929328856","heartbeat_front_addr":"172.21.15.114:6805/3929328856","state":["exists","up"]},{"osd":2,"uuid":"111f3ef1-97de-43aa-91a2-d7da6a2f7d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1364245230},{"type":"v1","addr":"172.21.15.177:6801","nonce":1364245230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1364245230},{"type":"v1","addr":"172.21.15.177:6803","nonce":1364245230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1364245230},{"type":"v1","addr":"172.21.15.177:6807","nonce":1364245230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1364245230},{"type":"v1","addr":"172.21.15.177:6805","nonce":1364245230}]},"public_addr":"172.21.15.177:6801/1364245230","cluster_addr":"172.21.15.177:6803/1364245230","heartbeat_back_addr":"172.21.15.177:6807/1364245230","heartbeat_front_addr":"172.21.15.177:6805/1364245230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:39:38.055997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:40:02.656316+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:40:27.326253+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/1018230916":"2024-03-27T04:37:58.646615+0000","172.21.15.38:6801/3938089582":"2024-03-27T04:37:58.646615+0000","172.21.15.38:0/4113838286":"2024-03-27T04:37:58.646615+0000","172.21.15.38:0/4140847":"2024-03-27T04:37:30.956666+0000","172.21.15.38:0/1308956235":"2024-03-27T04:37:30.956666+0000","172.21.15.38:0/378116029":"2024-03-27T04:37:30.956666+0000","172.21.15.38:6800/3938089582":"2024-03-27T04:37:58.646615+0000","172.21.15.38:6801/2491069991":"2024-03-27T04:37:30.956666+0000","172.21.15.38:6800/2491069991":"2024-03-27T04:37:30.956666+0000","172.21.15.38:0/329205786":"2024-03-27T04:37:58.646615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:41:10.581 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:10 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: ignoring --setuser ceph since I am not root 2024-03-26T04:41:10.581 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:10 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: ignoring --setgroup ceph since I am not root 2024-03-26T04:41:10.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:10 smithi038 ceph-mon[25836]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 2.8 KiB/s wr, 132 op/s 2024-03-26T04:41:10.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:10 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:41:10.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:10 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:41:10.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:10 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:41:10.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:10 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:41:10.645 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:10 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: ignoring --setuser ceph since I am not root 2024-03-26T04:41:10.645 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:10 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: ignoring --setgroup ceph since I am not root 2024-03-26T04:41:10.645 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:10 smithi114 ceph-mgr[29315]: -- 172.21.15.114:0/3502152548 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56402015e1a0 con 0x564020134800 2024-03-26T04:41:10.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:10 smithi114 ceph-mon[27627]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 2.8 KiB/s wr, 132 op/s 2024-03-26T04:41:10.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:10 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:41:10.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:10 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:41:10.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:10 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:41:10.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:10 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:41:10.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:10 smithi177 ceph-mon[27677]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 2.8 KiB/s wr, 132 op/s 2024-03-26T04:41:10.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:10 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:41:10.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:10 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:41:10.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:10 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' 2024-03-26T04:41:10.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:10 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:41:10.650 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:10 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: ignoring --setuser ceph since I am not root 2024-03-26T04:41:10.650 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:10 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: ignoring --setgroup ceph since I am not root 2024-03-26T04:41:10.650 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:10 smithi177 ceph-mgr[30146]: -- 172.21.15.177:0/1132052976 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c2242ac1a0 con 0x55c224282800 2024-03-26T04:41:11.330 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:10 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:10.959+0000 7fb716d69200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:41:11.331 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:11 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:11.057+0000 7fb716d69200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:41:11.354 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:10 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:10.911+0000 7ff061d86200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:41:11.354 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:11 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:11.003+0000 7ff061d86200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:41:11.354 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:10 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:10.918+0000 7f2c413d6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:41:11.354 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:11 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:11.010+0000 7f2c413d6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:41:11.641 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:11 smithi177 ceph-mon[27677]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:41:11.641 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:11 smithi177 ceph-mon[27677]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-26T04:41:11.641 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:11 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/672865372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:41:11.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:11 smithi114 ceph-mon[27627]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:41:11.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:11 smithi114 ceph-mon[27627]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-26T04:41:11.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:11 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/672865372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:41:11.666 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:11 smithi038 ceph-mon[25836]: from='mgr.14152 172.21.15.38:0/3026093399' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:41:11.666 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:11 smithi038 ceph-mon[25836]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-26T04:41:11.666 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:11 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/672865372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:41:11.899 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:11 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:11.640+0000 7f2c413d6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:41:11.899 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:11 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:11.736+0000 7f2c413d6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:41:11.899 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:11 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:11.884+0000 7f2c413d6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:41:11.928 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:11 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:11.636+0000 7ff061d86200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:41:11.928 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:11 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:11.731+0000 7ff061d86200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:41:11.972 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:11 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:11.731+0000 7fb716d69200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:41:11.972 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:11 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:11.828+0000 7fb716d69200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:41:11.973 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:11 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:11.972+0000 7fb716d69200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:41:12.062 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T04:41:12.062 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:41:12.231 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:11 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:11.870+0000 7ff061d86200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:41:12.231 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:11 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:11.965+0000 7ff061d86200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:41:12.231 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:12 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:12.047+0000 7ff061d86200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:41:12.310 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:12 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:12.084+0000 7fb716d69200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:41:12.310 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:12 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:12.173+0000 7fb716d69200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:41:12.343 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:11 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:11.985+0000 7f2c413d6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:41:12.344 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:12 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:12.074+0000 7f2c413d6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:41:12.344 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:12 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:12.343+0000 7f2c413d6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:41:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:12.507 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:12 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:12.311+0000 7ff061d86200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:41:12.507 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:12 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:12.420+0000 7ff061d86200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:41:12.507 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:12 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:12.506+0000 7ff061d86200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:41:12.562 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:12 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:12.476+0000 7fb716d69200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:41:12.635 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:12 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:12.453+0000 7f2c413d6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:41:12.635 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:12 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:12.539+0000 7f2c413d6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:41:12.635 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:12 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:12.634+0000 7f2c413d6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:41:12.778 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:12 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:12.603+0000 7ff061d86200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:41:12.830 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:12 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:12.595+0000 7fb716d69200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:41:12.830 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:12 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:12.691+0000 7fb716d69200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:41:12.831 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:12 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:12.794+0000 7fb716d69200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:41:12.899 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:12 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:12.812+0000 7f2c413d6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:41:13.132 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:12 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:12.982+0000 7fb716d69200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:41:13.145 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:12 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:12.776+0000 7ff061d86200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:41:13.895 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:13 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:13.746+0000 7ff061d86200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:41:13.895 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:13 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:13.880+0000 7ff061d86200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:41:14.149 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:13.764+0000 7f2c413d6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:41:14.149 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:13.891+0000 7f2c413d6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:41:14.173 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:13 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:13.979+0000 7fb716d69200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:41:14.173 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:14 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:14.115+0000 7fb716d69200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:41:14.504 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:14 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:14.223+0000 7f2c413d6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:41:14.504 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:14 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:14.311+0000 7f2c413d6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:41:14.504 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:14 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:14.503+0000 7f2c413d6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:41:14.575 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:14 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:14.223+0000 7ff061d86200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:41:14.575 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:14 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:14.329+0000 7ff061d86200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:41:14.575 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:14 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:14.575+0000 7ff061d86200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:41:14.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:14 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:14.465+0000 7fb716d69200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:41:14.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:14 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:14.561+0000 7fb716d69200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:41:14.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:14 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:14.775+0000 7fb716d69200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:41:14.895 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:14 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:14.793+0000 7ff061d86200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:41:14.899 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:14 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:14.683+0000 7f2c413d6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:41:14.906 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:41:14.906 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":30,"fsid":"56caa506-eb2a-11ee-95d0-87774f69a715","created":"2024-03-26T04:36:55.633203+0000","modified":"2024-03-26T04:40:56.017556+0000","last_up_change":"2024-03-26T04:40:29.706706+0000","last_in_change":"2024-03-26T04:40:10.202973+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:40:30.996140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T04:40:47.274735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-26T04:40:49.303156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-26T04:40:51.264415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-26T04:40:53.320116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"36825eb5-ff6e-4e19-adab-791266b00c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":755874714},{"type":"v1","addr":"172.21.15.38:6803","nonce":755874714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":755874714},{"type":"v1","addr":"172.21.15.38:6805","nonce":755874714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":755874714},{"type":"v1","addr":"172.21.15.38:6809","nonce":755874714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":755874714},{"type":"v1","addr":"172.21.15.38:6807","nonce":755874714}]},"public_addr":"172.21.15.38:6803/755874714","cluster_addr":"172.21.15.38:6805/755874714","heartbeat_back_addr":"172.21.15.38:6809/755874714","heartbeat_front_addr":"172.21.15.38:6807/755874714","state":["exists","up"]},{"osd":1,"uuid":"631b8290-ee85-4525-be34-13e99b5e9573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3929328856},{"type":"v1","addr":"172.21.15.114:6801","nonce":3929328856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3929328856},{"type":"v1","addr":"172.21.15.114:6803","nonce":3929328856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3929328856},{"type":"v1","addr":"172.21.15.114:6807","nonce":3929328856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3929328856},{"type":"v1","addr":"172.21.15.114:6805","nonce":3929328856}]},"public_addr":"172.21.15.114:6801/3929328856","cluster_addr":"172.21.15.114:6803/3929328856","heartbeat_back_addr":"172.21.15.114:6807/3929328856","heartbeat_front_addr":"172.21.15.114:6805/3929328856","state":["exists","up"]},{"osd":2,"uuid":"111f3ef1-97de-43aa-91a2-d7da6a2f7d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1364245230},{"type":"v1","addr":"172.21.15.177:6801","nonce":1364245230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1364245230},{"type":"v1","addr":"172.21.15.177:6803","nonce":1364245230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1364245230},{"type":"v1","addr":"172.21.15.177:6807","nonce":1364245230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1364245230},{"type":"v1","addr":"172.21.15.177:6805","nonce":1364245230}]},"public_addr":"172.21.15.177:6801/1364245230","cluster_addr":"172.21.15.177:6803/1364245230","heartbeat_back_addr":"172.21.15.177:6807/1364245230","heartbeat_front_addr":"172.21.15.177:6805/1364245230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:39:38.055997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:40:02.656316+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:40:27.326253+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/1018230916":"2024-03-27T04:37:58.646615+0000","172.21.15.38:6801/3938089582":"2024-03-27T04:37:58.646615+0000","172.21.15.38:0/4113838286":"2024-03-27T04:37:58.646615+0000","172.21.15.38:0/4140847":"2024-03-27T04:37:30.956666+0000","172.21.15.38:0/1308956235":"2024-03-27T04:37:30.956666+0000","172.21.15.38:0/378116029":"2024-03-27T04:37:30.956666+0000","172.21.15.38:6800/3938089582":"2024-03-27T04:37:58.646615+0000","172.21.15.38:6801/2491069991":"2024-03-27T04:37:30.956666+0000","172.21.15.38:6800/2491069991":"2024-03-27T04:37:30.956666+0000","172.21.15.38:0/329205786":"2024-03-27T04:37:58.646615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:41:15.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:14 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/3823213881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:41:15.081 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:14 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:14.973+0000 7fb716d69200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:41:15.226 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:14 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/3823213881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:41:15.227 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:14 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:14.965+0000 7f2c413d6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:41:15.251 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:14 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/3823213881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:41:15.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:15 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:15.089+0000 7ff061d86200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:41:15.543 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:15 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:15.279+0000 7fb716d69200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:41:15.597 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T04:41:15.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T04:41:15.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T04:41:15.604 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:15 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:15.350+0000 7ff061d86200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:41:15.649 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:15 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:15.226+0000 7f2c413d6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:41:15.649 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:15 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:15.461+0000 7f2c413d6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:41:15.830 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:15 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:15.542+0000 7fb716d69200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:41:15.869 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:15 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:15.604+0000 7ff061d86200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:41:15.870 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:15.873 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:16.100 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:16.125 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:15 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:15.866+0000 7fb716d69200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:41:16.645 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:16 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:16.446+0000 7ff061d86200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:41:16.649 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:16 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:16.250+0000 7f2c413d6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:41:17.056 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:16 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:16.777+0000 7fb716d69200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:41:17.125 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:16 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:16.814+0000 7f2c413d6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:41:17.126 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:16 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:16.910+0000 7f2c413d6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:41:17.126 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:17 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:17.125+0000 7f2c413d6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:41:17.320 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:17.006+0000 7ff061d86200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:41:17.321 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:17.104+0000 7ff061d86200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:41:17.321 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-mon[27627]: Standby manager daemon z restarted 2024-03-26T04:41:17.321 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-mon[27627]: Standby manager daemon z started 2024-03-26T04:41:17.321 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.177:0/2873862232' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T04:41:17.321 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.177:0/2873862232' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:17.321 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.177:0/2873862232' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T04:41:17.321 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.177:0/2873862232' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:17.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-mon[25836]: Standby manager daemon z restarted 2024-03-26T04:41:17.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-mon[25836]: Standby manager daemon z started 2024-03-26T04:41:17.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.177:0/2873862232' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T04:41:17.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.177:0/2873862232' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:17.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.177:0/2873862232' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T04:41:17.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.177:0/2873862232' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:17.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:17 smithi177 ceph-mon[27677]: Standby manager daemon z restarted 2024-03-26T04:41:17.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:17 smithi177 ceph-mon[27677]: Standby manager daemon z started 2024-03-26T04:41:17.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:17 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.177:0/2873862232' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T04:41:17.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:17 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.177:0/2873862232' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:17.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:17 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.177:0/2873862232' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T04:41:17.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:17 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.177:0/2873862232' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:17.400 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:17 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: [26/Mar/2024:04:41:17] ENGINE Bus STARTING 2024-03-26T04:41:17.400 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:17 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: CherryPy Checker: 2024-03-26T04:41:17.400 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:17 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: The Application mounted at '' has an empty config. 2024-03-26T04:41:17.400 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:17 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: 2024-03-26T04:41:17.400 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:17 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: [26/Mar/2024:04:41:17] ENGINE Serving on http://:::9283 2024-03-26T04:41:17.400 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:41:17 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z[30073]: [26/Mar/2024:04:41:17] ENGINE Bus STARTED 2024-03-26T04:41:17.583 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:17.368+0000 7fb716d69200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:41:17.583 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:17.477+0000 7fb716d69200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:41:17.645 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:17.320+0000 7ff061d86200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:41:17.645 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: [26/Mar/2024:04:41:17] ENGINE Bus STARTING 2024-03-26T04:41:17.646 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: CherryPy Checker: 2024-03-26T04:41:17.646 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: The Application mounted at '' has an empty config. 2024-03-26T04:41:17.646 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: 2024-03-26T04:41:17.646 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: [26/Mar/2024:04:41:17] ENGINE Serving on http://:::9283 2024-03-26T04:41:17.646 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:41:17 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x[29292]: [26/Mar/2024:04:41:17] ENGINE Bus STARTED 2024-03-26T04:41:17.885 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:17.713+0000 7fb716d69200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:41:17.885 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:41:17] ENGINE Bus STARTING 2024-03-26T04:41:18.203 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: CherryPy Checker: 2024-03-26T04:41:18.203 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: The Application mounted at '' has an empty config. 2024-03-26T04:41:18.203 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:41:18.203 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:41:17] ENGINE Serving on http://:::9283 2024-03-26T04:41:18.203 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:17 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:41:17] ENGINE Bus STARTED 2024-03-26T04:41:18.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-26T04:41:18.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: Standby manager daemon x restarted 2024-03-26T04:41:18.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: Standby manager daemon x started 2024-03-26T04:41:18.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.114:0/3976373813' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:41:18.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.114:0/3976373813' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:18.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.114:0/3976373813' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:41:18.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.? 172.21.15.114:0/3976373813' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:18.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: Active manager daemon y restarted 2024-03-26T04:41:18.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: Activating manager daemon y 2024-03-26T04:41:18.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: osdmap e31: 3 total, 3 up, 3 in 2024-03-26T04:41:18.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: mgrmap e19: y(active, starting, since 0.0128551s), standbys: x, z 2024-03-26T04:41:18.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:41:18.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:18.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:18.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:41:18.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:41:18.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T04:41:18.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:41:18.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:41:18.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:41:18.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:41:18.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:41:18.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:41:18.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: Manager daemon y is now available 2024-03-26T04:41:18.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:18.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:18.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:18.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:41:18.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:41:18.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:41:18.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]': finished 2024-03-26T04:41:18.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]': finished 2024-03-26T04:41:18.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]': finished 2024-03-26T04:41:18.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:18.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:18.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:18.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:18.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:41:18.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:41:18.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-26T04:41:18.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: Standby manager daemon x restarted 2024-03-26T04:41:18.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: Standby manager daemon x started 2024-03-26T04:41:18.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.114:0/3976373813' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:41:18.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.114:0/3976373813' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.114:0/3976373813' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.? 172.21.15.114:0/3976373813' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: Active manager daemon y restarted 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: Activating manager daemon y 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: osdmap e31: 3 total, 3 up, 3 in 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: mgrmap e19: y(active, starting, since 0.0128551s), standbys: x, z 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:41:18.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: Manager daemon y is now available 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]': finished 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]': finished 2024-03-26T04:41:18.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]': finished 2024-03-26T04:41:18.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:18.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:18.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:18.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:18.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:41:18.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:41:18.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-26T04:41:18.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: Standby manager daemon x restarted 2024-03-26T04:41:18.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: Standby manager daemon x started 2024-03-26T04:41:18.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.114:0/3976373813' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:41:18.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.114:0/3976373813' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:18.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.114:0/3976373813' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:41:18.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.? 172.21.15.114:0/3976373813' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:18.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: Active manager daemon y restarted 2024-03-26T04:41:18.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: Activating manager daemon y 2024-03-26T04:41:18.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: osdmap e31: 3 total, 3 up, 3 in 2024-03-26T04:41:18.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: mgrmap e19: y(active, starting, since 0.0128551s), standbys: x, z 2024-03-26T04:41:18.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:41:18.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:18.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:18.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:41:18.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:41:18.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T04:41:18.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:41:18.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:41:18.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:41:18.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:41:18.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: Manager daemon y is now available 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]': finished 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]': finished 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]': finished 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:18.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:41:18.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:41:19.418 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:19 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: ::ffff:172.21.15.114 - - [26/Mar/2024:04:41:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:41:19.418 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:19 smithi038 ceph-mon[25836]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T04:41:19.419 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:19 smithi038 ceph-mon[25836]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-26T04:41:19.419 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:19 smithi038 ceph-mon[25836]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-26T04:41:19.419 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:19 smithi038 ceph-mon[25836]: Updating smithi038:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:19.419 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:19 smithi038 ceph-mon[25836]: [26/Mar/2024:04:41:18] ENGINE Bus STARTING 2024-03-26T04:41:19.419 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:19 smithi038 ceph-mon[25836]: mgrmap e20: y(active, since 1.02307s), standbys: x, z 2024-03-26T04:41:19.419 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:19 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:19.419 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:19 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:19.419 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:19 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:19.419 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:19 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:19.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:19 smithi177 ceph-mon[27677]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T04:41:19.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:19 smithi177 ceph-mon[27677]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-26T04:41:19.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:19 smithi177 ceph-mon[27677]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-26T04:41:19.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:19 smithi177 ceph-mon[27677]: Updating smithi038:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:19.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:19 smithi177 ceph-mon[27677]: [26/Mar/2024:04:41:18] ENGINE Bus STARTING 2024-03-26T04:41:19.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:19 smithi177 ceph-mon[27677]: mgrmap e20: y(active, since 1.02307s), standbys: x, z 2024-03-26T04:41:19.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:19 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:19.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:19 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:19.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:19 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:19.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:19 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:19.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:19 smithi114 ceph-mon[27627]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T04:41:19.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:19 smithi114 ceph-mon[27627]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-26T04:41:19.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:19 smithi114 ceph-mon[27627]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-26T04:41:19.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:19 smithi114 ceph-mon[27627]: Updating smithi038:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:19.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:19 smithi114 ceph-mon[27627]: [26/Mar/2024:04:41:18] ENGINE Bus STARTING 2024-03-26T04:41:19.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:19 smithi114 ceph-mon[27627]: mgrmap e20: y(active, since 1.02307s), standbys: x, z 2024-03-26T04:41:19.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:19 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:19.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:19 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:19.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:19 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:19.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:19 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:20.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: Updating smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:20.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: Updating smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:20.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: [26/Mar/2024:04:41:18] ENGINE Serving on http://172.21.15.38:8765 2024-03-26T04:41:20.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:20.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: [26/Mar/2024:04:41:18] ENGINE Serving on https://172.21.15.38:7151 2024-03-26T04:41:20.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: [26/Mar/2024:04:41:18] ENGINE Bus STARTED 2024-03-26T04:41:20.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:20.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:20.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:20.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: Updating smithi038:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:41:20.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: Updating smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:41:20.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: Updating smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:41:20.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:20.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:20.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:20.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:41:20.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:20.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:20 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi038", "caps": []}]: dispatch 2024-03-26T04:41:20.453 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: Updating smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:20.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: Updating smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:20.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: [26/Mar/2024:04:41:18] ENGINE Serving on http://172.21.15.38:8765 2024-03-26T04:41:20.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:20.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: [26/Mar/2024:04:41:18] ENGINE Serving on https://172.21.15.38:7151 2024-03-26T04:41:20.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: [26/Mar/2024:04:41:18] ENGINE Bus STARTED 2024-03-26T04:41:20.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:20.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:20.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:20.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: Updating smithi038:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:41:20.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: Updating smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:41:20.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: Updating smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:41:20.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:20.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:20.455 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:20.455 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:41:20.455 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:20.455 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:20 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi038", "caps": []}]: dispatch 2024-03-26T04:41:20.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: Updating smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: Updating smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: [26/Mar/2024:04:41:18] ENGINE Serving on http://172.21.15.38:8765 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: [26/Mar/2024:04:41:18] ENGINE Serving on https://172.21.15.38:7151 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: [26/Mar/2024:04:41:18] ENGINE Bus STARTED 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: Updating smithi038:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: Updating smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: Updating smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:20.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:20.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:41:20.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:20.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:20 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi038", "caps": []}]: dispatch 2024-03-26T04:41:21.295 INFO:teuthology.orchestra.run.smithi038.stdout:34359738390 2024-03-26T04:41:21.295 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T04:41:21.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:21 smithi038 ceph-mon[25836]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:21.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:21 smithi038 ceph-mon[25836]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-03-26T04:41:21.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:21 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi114", "caps": []}]: dispatch 2024-03-26T04:41:21.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:21 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]: dispatch 2024-03-26T04:41:21.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:21 smithi114 ceph-mon[27627]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:21.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:21 smithi114 ceph-mon[27627]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-03-26T04:41:21.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:21 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi114", "caps": []}]: dispatch 2024-03-26T04:41:21.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:21 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]: dispatch 2024-03-26T04:41:21.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:21 smithi177 ceph-mon[27677]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:21.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:21 smithi177 ceph-mon[27677]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-03-26T04:41:21.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:21 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi114", "caps": []}]: dispatch 2024-03-26T04:41:21.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:21 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]: dispatch 2024-03-26T04:41:21.740 INFO:teuthology.orchestra.run.smithi038.stdout:77309411340 2024-03-26T04:41:21.741 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T04:41:22.412 INFO:teuthology.orchestra.run.smithi038.stdout:55834574864 2024-03-26T04:41:22.412 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T04:41:22.508 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:22.516 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:23.296 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:23 smithi114 ceph-mon[27627]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-26T04:41:23.297 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:23 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.297 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:23 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.297 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:23 smithi114 ceph-mon[27627]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-26T04:41:23.297 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:23 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:23.297 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:23 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi038", "caps": []}]: dispatch 2024-03-26T04:41:23.297 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:23 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]: dispatch 2024-03-26T04:41:23.297 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:23 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.297 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:23 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.297 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:23 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.297 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:23 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.311 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:23 smithi038 ceph-mon[25836]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-26T04:41:23.312 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:23 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.312 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:23 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.312 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:23 smithi038 ceph-mon[25836]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-26T04:41:23.312 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:23 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:23.312 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:23 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi038", "caps": []}]: dispatch 2024-03-26T04:41:23.312 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:23 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]: dispatch 2024-03-26T04:41:23.312 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:23 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.312 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:23 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.312 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:23 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.312 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:23 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:23 smithi177 ceph-mon[27677]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-26T04:41:23.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:23 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:23 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:23 smithi177 ceph-mon[27677]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-26T04:41:23.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:23 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:23.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:23 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi038", "caps": []}]: dispatch 2024-03-26T04:41:23.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:23 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]: dispatch 2024-03-26T04:41:23.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:23 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:23 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:23 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:23.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:23 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.132 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:24 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.132 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:24 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.132 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:24 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:24.132 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:24 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:24.132 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:24 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.132 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:24 smithi114 ceph-mon[27627]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:41:24.132 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:24 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.133 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:24 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.133 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:24 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.133 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:24 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]: dispatch 2024-03-26T04:41:24.133 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:24 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.328 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:24 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.329 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:24 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.329 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:24 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:24.329 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:24 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:24.329 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:24 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.329 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:24 smithi038 ceph-mon[25836]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:41:24.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:24 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:24 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:24 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:24 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]: dispatch 2024-03-26T04:41:24.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:24 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.377 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:24 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:24 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:24 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:24.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:24 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:24.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:24 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:24 smithi177 ceph-mon[27677]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:41:24.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:24 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:24 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:24 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:24.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:24 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]: dispatch 2024-03-26T04:41:24.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:24 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-26T04:41:25.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:25 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-26T04:41:25.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:25 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-26T04:41:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:25 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:26 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:26 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:26 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:26 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:26 smithi038 ceph-mon[25836]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T04:41:26.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:26 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:26 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:26 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.798 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:26 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.799 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:26 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.799 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:26 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.799 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:26 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.799 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:26 smithi114 ceph-mon[27627]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T04:41:26.799 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:26 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.799 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:26 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.799 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:26 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.812 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:26 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.812 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:26 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.813 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:26 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.813 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:26 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.813 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:26 smithi177 ceph-mon[27677]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T04:41:26.813 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:26 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.813 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:26 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:26.813 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:26 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:27.213 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:27.441 INFO:teuthology.orchestra.run.smithi038.stdout:34359738391 2024-03-26T04:41:27.775 INFO:teuthology.orchestra.run.smithi038.stdout:77309411341 2024-03-26T04:41:28.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:27 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:27 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:27 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:27 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2367394507' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:41:28.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:27 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:27 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:27 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:27 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/2838143911' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:41:28.212 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:27 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.212 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:27 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.212 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:27 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.212 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:27 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2367394507' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:41:28.212 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:27 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.212 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:27 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.212 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:27 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.212 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:27 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/2838143911' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:41:28.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:27 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:27 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:27 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:27 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2367394507' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:41:28.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:27 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:27 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:27 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:28.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:27 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/2838143911' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:41:29.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:29 smithi177 ceph-mon[27677]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T04:41:29.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:29 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:29.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:29 smithi038 ceph-mon[25836]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T04:41:29.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:29 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:29.417 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:29 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: ::ffff:172.21.15.114 - - [26/Mar/2024:04:41:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:29 smithi114 ceph-mon[27627]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:29 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:29.801 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738390 got 34359738391 for osd.0 2024-03-26T04:41:29.801 DEBUG:teuthology.parallel:result is None 2024-03-26T04:41:30.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:30 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:30 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:30 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:30 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:30 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:30 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:30 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:30.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:30 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:30.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:30 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:30.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:30 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.548 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411340 got 77309411341 for osd.2 2024-03-26T04:41:30.548 DEBUG:teuthology.parallel:result is None 2024-03-26T04:41:30.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:30 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:30 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:30 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:30 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:30 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:30 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:30 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:30.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:30 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:30.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:30 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:30.897 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:30 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:30 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:30 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:30 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:30 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:30 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:30 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:30.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:30 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:30.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:30 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:30.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:30 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:30.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:30 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:31.525 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:31 smithi038 ceph-mon[25836]: Deploying daemon alertmanager.a on smithi038 2024-03-26T04:41:31.525 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:31 smithi038 ceph-mon[25836]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:41:31.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:31 smithi114 ceph-mon[27627]: Deploying daemon alertmanager.a on smithi038 2024-03-26T04:41:31.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:31 smithi114 ceph-mon[27627]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:41:31.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:31 smithi177 ceph-mon[27677]: Deploying daemon alertmanager.a on smithi038 2024-03-26T04:41:31.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:31 smithi177 ceph-mon[27677]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:41:32.126 INFO:teuthology.orchestra.run.smithi038.stdout:55834574866 2024-03-26T04:41:32.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:32 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/385203455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:41:32.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:32 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/385203455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:41:32.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:32 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/385203455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:41:33.102 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574864 got 55834574866 for osd.1 2024-03-26T04:41:33.102 DEBUG:teuthology.parallel:result is None 2024-03-26T04:41:33.102 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T04:41:33.102 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:41:33.344 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:33.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:33 smithi038 ceph-mon[25836]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T04:41:33.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:33 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:33.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:33 smithi114 ceph-mon[27627]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T04:41:33.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:33 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:33.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:33 smithi177 ceph-mon[27677]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T04:41:33.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:33 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:34.700 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:34 smithi038 ceph-mon[25836]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:34.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:34 smithi114 ceph-mon[27627]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:34.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:34 smithi177 ceph-mon[27677]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:36.542 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:41:36.544 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-26T04:41:36.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:36 smithi038 ceph-mon[25836]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:37.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:36 smithi114 ceph-mon[27627]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:37.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:36 smithi177 ceph-mon[27677]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:37.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:37 smithi038 ceph-mon[25836]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:41:38.018 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-03-26T04:41:35.737000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96612,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126556,"statfs":{"total":287972524032,"available":287873593344,"internally_reserved":0,"allocated":3158016,"data_stored":2100351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002506"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725122+0000","last_change":"2024-03-26T04:40:55.035961+0000","last_active":"2024-03-26T04:41:17.725122+0000","last_peered":"2024-03-26T04:41:17.725122+0000","last_clean":"2024-03-26T04:41:17.725122+0000","last_became_active":"2024-03-26T04:40:55.034664+0000","last_became_peered":"2024-03-26T04:40:55.034664+0000","last_unstale":"2024-03-26T04:41:17.725122+0000","last_undegraded":"2024-03-26T04:41:17.725122+0000","last_fullsized":"2024-03-26T04:41:17.725122+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:19:56.073034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724259+0000","last_change":"2024-03-26T04:40:53.013211+0000","last_active":"2024-03-26T04:41:17.724259+0000","last_peered":"2024-03-26T04:41:17.724259+0000","last_clean":"2024-03-26T04:41:17.724259+0000","last_became_active":"2024-03-26T04:40:53.011165+0000","last_became_peered":"2024-03-26T04:40:53.011165+0000","last_unstale":"2024-03-26T04:41:17.724259+0000","last_undegraded":"2024-03-26T04:41:17.724259+0000","last_fullsized":"2024-03-26T04:41:17.724259+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:16:25.123449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726192+0000","last_change":"2024-03-26T04:40:48.985731+0000","last_active":"2024-03-26T04:41:17.726192+0000","last_peered":"2024-03-26T04:41:17.726192+0000","last_clean":"2024-03-26T04:41:17.726192+0000","last_became_active":"2024-03-26T04:40:48.985418+0000","last_became_peered":"2024-03-26T04:40:48.985418+0000","last_unstale":"2024-03-26T04:41:17.726192+0000","last_undegraded":"2024-03-26T04:41:17.726192+0000","last_fullsized":"2024-03-26T04:41:17.726192+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:26:03.466720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724762+0000","last_change":"2024-03-26T04:40:51.005807+0000","last_active":"2024-03-26T04:41:17.724762+0000","last_peered":"2024-03-26T04:41:17.724762+0000","last_clean":"2024-03-26T04:41:17.724762+0000","last_became_active":"2024-03-26T04:40:51.004795+0000","last_became_peered":"2024-03-26T04:40:51.004795+0000","last_unstale":"2024-03-26T04:41:17.724762+0000","last_undegraded":"2024-03-26T04:41:17.724762+0000","last_fullsized":"2024-03-26T04:41:17.724762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:19:54.426252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723243+0000","last_change":"2024-03-26T04:40:55.035961+0000","last_active":"2024-03-26T04:41:17.723243+0000","last_peered":"2024-03-26T04:41:17.723243+0000","last_clean":"2024-03-26T04:41:17.723243+0000","last_became_active":"2024-03-26T04:40:55.035286+0000","last_became_peered":"2024-03-26T04:40:55.035286+0000","last_unstale":"2024-03-26T04:41:17.723243+0000","last_undegraded":"2024-03-26T04:41:17.723243+0000","last_fullsized":"2024-03-26T04:41:17.723243+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:25:54.077476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724811+0000","last_change":"2024-03-26T04:40:53.015612+0000","last_active":"2024-03-26T04:41:17.724811+0000","last_peered":"2024-03-26T04:41:17.724811+0000","last_clean":"2024-03-26T04:41:17.724811+0000","last_became_active":"2024-03-26T04:40:53.015267+0000","last_became_peered":"2024-03-26T04:40:53.015267+0000","last_unstale":"2024-03-26T04:41:17.724811+0000","last_undegraded":"2024-03-26T04:41:17.724811+0000","last_fullsized":"2024-03-26T04:41:17.724811+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:02:55.971241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725436+0000","last_change":"2024-03-26T04:40:48.990625+0000","last_active":"2024-03-26T04:41:17.725436+0000","last_peered":"2024-03-26T04:41:17.725436+0000","last_clean":"2024-03-26T04:41:17.725436+0000","last_became_active":"2024-03-26T04:40:48.989155+0000","last_became_peered":"2024-03-26T04:40:48.989155+0000","last_unstale":"2024-03-26T04:41:17.725436+0000","last_undegraded":"2024-03-26T04:41:17.725436+0000","last_fullsized":"2024-03-26T04:41:17.725436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:53:32.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725961+0000","last_change":"2024-03-26T04:40:51.005802+0000","last_active":"2024-03-26T04:41:17.725961+0000","last_peered":"2024-03-26T04:41:17.725961+0000","last_clean":"2024-03-26T04:41:17.725961+0000","last_became_active":"2024-03-26T04:40:51.004757+0000","last_became_peered":"2024-03-26T04:40:51.004757+0000","last_unstale":"2024-03-26T04:41:17.725961+0000","last_undegraded":"2024-03-26T04:41:17.725961+0000","last_fullsized":"2024-03-26T04:41:17.725961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:58:05.869971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724540+0000","last_change":"2024-03-26T04:40:55.032958+0000","last_active":"2024-03-26T04:41:17.724540+0000","last_peered":"2024-03-26T04:41:17.724540+0000","last_clean":"2024-03-26T04:41:17.724540+0000","last_became_active":"2024-03-26T04:40:55.032654+0000","last_became_peered":"2024-03-26T04:40:55.032654+0000","last_unstale":"2024-03-26T04:41:17.724540+0000","last_undegraded":"2024-03-26T04:41:17.724540+0000","last_fullsized":"2024-03-26T04:41:17.724540+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:44:43.531957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725690+0000","last_change":"2024-03-26T04:40:53.015200+0000","last_active":"2024-03-26T04:41:17.725690+0000","last_peered":"2024-03-26T04:41:17.725690+0000","last_clean":"2024-03-26T04:41:17.725690+0000","last_became_active":"2024-03-26T04:40:53.014211+0000","last_became_peered":"2024-03-26T04:40:53.014211+0000","last_unstale":"2024-03-26T04:41:17.725690+0000","last_undegraded":"2024-03-26T04:41:17.725690+0000","last_fullsized":"2024-03-26T04:41:17.725690+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:04:47.693417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727617+0000","last_change":"2024-03-26T04:40:48.989466+0000","last_active":"2024-03-26T04:41:17.727617+0000","last_peered":"2024-03-26T04:41:17.727617+0000","last_clean":"2024-03-26T04:41:17.727617+0000","last_became_active":"2024-03-26T04:40:48.989157+0000","last_became_peered":"2024-03-26T04:40:48.989157+0000","last_unstale":"2024-03-26T04:41:17.727617+0000","last_undegraded":"2024-03-26T04:41:17.727617+0000","last_fullsized":"2024-03-26T04:41:17.727617+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:48:47.660575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725653+0000","last_change":"2024-03-26T04:40:51.004199+0000","last_active":"2024-03-26T04:41:17.725653+0000","last_peered":"2024-03-26T04:41:17.725653+0000","last_clean":"2024-03-26T04:41:17.725653+0000","last_became_active":"2024-03-26T04:40:51.003398+0000","last_became_peered":"2024-03-26T04:40:51.003398+0000","last_unstale":"2024-03-26T04:41:17.725653+0000","last_undegraded":"2024-03-26T04:41:17.725653+0000","last_fullsized":"2024-03-26T04:41:17.725653+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:21:17.714545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725450+0000","last_change":"2024-03-26T04:40:55.034310+0000","last_active":"2024-03-26T04:41:17.725450+0000","last_peered":"2024-03-26T04:41:17.725450+0000","last_clean":"2024-03-26T04:41:17.725450+0000","last_became_active":"2024-03-26T04:40:55.033877+0000","last_became_peered":"2024-03-26T04:40:55.033877+0000","last_unstale":"2024-03-26T04:41:17.725450+0000","last_undegraded":"2024-03-26T04:41:17.725450+0000","last_fullsized":"2024-03-26T04:41:17.725450+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:59:59.353480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724729+0000","last_change":"2024-03-26T04:40:53.014572+0000","last_active":"2024-03-26T04:41:17.724729+0000","last_peered":"2024-03-26T04:41:17.724729+0000","last_clean":"2024-03-26T04:41:17.724729+0000","last_became_active":"2024-03-26T04:40:53.014176+0000","last_became_peered":"2024-03-26T04:40:53.014176+0000","last_unstale":"2024-03-26T04:41:17.724729+0000","last_undegraded":"2024-03-26T04:41:17.724729+0000","last_fullsized":"2024-03-26T04:41:17.724729+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:12:52.085798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726193+0000","last_change":"2024-03-26T04:40:48.989578+0000","last_active":"2024-03-26T04:41:17.726193+0000","last_peered":"2024-03-26T04:41:17.726193+0000","last_clean":"2024-03-26T04:41:17.726193+0000","last_became_active":"2024-03-26T04:40:48.989207+0000","last_became_peered":"2024-03-26T04:40:48.989207+0000","last_unstale":"2024-03-26T04:41:17.726193+0000","last_undegraded":"2024-03-26T04:41:17.726193+0000","last_fullsized":"2024-03-26T04:41:17.726193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:00:54.968707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'18","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727517+0000","last_change":"2024-03-26T04:40:51.008169+0000","last_active":"2024-03-26T04:41:17.727517+0000","last_peered":"2024-03-26T04:41:17.727517+0000","last_clean":"2024-03-26T04:41:17.727517+0000","last_became_active":"2024-03-26T04:40:51.007963+0000","last_became_peered":"2024-03-26T04:40:51.007963+0000","last_unstale":"2024-03-26T04:41:17.727517+0000","last_undegraded":"2024-03-26T04:41:17.727517+0000","last_fullsized":"2024-03-26T04:41:17.727517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:01:08.710764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725299+0000","last_change":"2024-03-26T04:40:55.035526+0000","last_active":"2024-03-26T04:41:17.725299+0000","last_peered":"2024-03-26T04:41:17.725299+0000","last_clean":"2024-03-26T04:41:17.725299+0000","last_became_active":"2024-03-26T04:40:55.034017+0000","last_became_peered":"2024-03-26T04:40:55.034017+0000","last_unstale":"2024-03-26T04:41:17.725299+0000","last_undegraded":"2024-03-26T04:41:17.725299+0000","last_fullsized":"2024-03-26T04:41:17.725299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:18:23.243089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726705+0000","last_change":"2024-03-26T04:40:53.018218+0000","last_active":"2024-03-26T04:41:17.726705+0000","last_peered":"2024-03-26T04:41:17.726705+0000","last_clean":"2024-03-26T04:41:17.726705+0000","last_became_active":"2024-03-26T04:40:53.017928+0000","last_became_peered":"2024-03-26T04:40:53.017928+0000","last_unstale":"2024-03-26T04:41:17.726705+0000","last_undegraded":"2024-03-26T04:41:17.726705+0000","last_fullsized":"2024-03-26T04:41:17.726705+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:30:56.831646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725528+0000","last_change":"2024-03-26T04:40:48.989361+0000","last_active":"2024-03-26T04:41:17.725528+0000","last_peered":"2024-03-26T04:41:17.725528+0000","last_clean":"2024-03-26T04:41:17.725528+0000","last_became_active":"2024-03-26T04:40:48.988913+0000","last_became_peered":"2024-03-26T04:40:48.988913+0000","last_unstale":"2024-03-26T04:41:17.725528+0000","last_undegraded":"2024-03-26T04:41:17.725528+0000","last_fullsized":"2024-03-26T04:41:17.725528+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:42:23.174794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725552+0000","last_change":"2024-03-26T04:40:51.009139+0000","last_active":"2024-03-26T04:41:17.725552+0000","last_peered":"2024-03-26T04:41:17.725552+0000","last_clean":"2024-03-26T04:41:17.725552+0000","last_became_active":"2024-03-26T04:40:51.008851+0000","last_became_peered":"2024-03-26T04:40:51.008851+0000","last_unstale":"2024-03-26T04:41:17.725552+0000","last_undegraded":"2024-03-26T04:41:17.725552+0000","last_fullsized":"2024-03-26T04:41:17.725552+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:14:51.880865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725812+0000","last_change":"2024-03-26T04:40:55.036209+0000","last_active":"2024-03-26T04:41:17.725812+0000","last_peered":"2024-03-26T04:41:17.725812+0000","last_clean":"2024-03-26T04:41:17.725812+0000","last_became_active":"2024-03-26T04:40:55.035869+0000","last_became_peered":"2024-03-26T04:40:55.035869+0000","last_unstale":"2024-03-26T04:41:17.725812+0000","last_undegraded":"2024-03-26T04:41:17.725812+0000","last_fullsized":"2024-03-26T04:41:17.725812+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:19:29.554079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726029+0000","last_change":"2024-03-26T04:40:53.012803+0000","last_active":"2024-03-26T04:41:17.726029+0000","last_peered":"2024-03-26T04:41:17.726029+0000","last_clean":"2024-03-26T04:41:17.726029+0000","last_became_active":"2024-03-26T04:40:53.011029+0000","last_became_peered":"2024-03-26T04:40:53.011029+0000","last_unstale":"2024-03-26T04:41:17.726029+0000","last_undegraded":"2024-03-26T04:41:17.726029+0000","last_fullsized":"2024-03-26T04:41:17.726029+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:40:32.531037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726012+0000","last_change":"2024-03-26T04:40:48.986540+0000","last_active":"2024-03-26T04:41:17.726012+0000","last_peered":"2024-03-26T04:41:17.726012+0000","last_clean":"2024-03-26T04:41:17.726012+0000","last_became_active":"2024-03-26T04:40:48.985152+0000","last_became_peered":"2024-03-26T04:40:48.985152+0000","last_unstale":"2024-03-26T04:41:17.726012+0000","last_undegraded":"2024-03-26T04:41:17.726012+0000","last_fullsized":"2024-03-26T04:41:17.726012+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:36:47.500789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725211+0000","last_change":"2024-03-26T04:40:51.009795+0000","last_active":"2024-03-26T04:41:17.725211+0000","last_peered":"2024-03-26T04:41:17.725211+0000","last_clean":"2024-03-26T04:41:17.725211+0000","last_became_active":"2024-03-26T04:40:51.009439+0000","last_became_peered":"2024-03-26T04:40:51.009439+0000","last_unstale":"2024-03-26T04:41:17.725211+0000","last_undegraded":"2024-03-26T04:41:17.725211+0000","last_fullsized":"2024-03-26T04:41:17.725211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:13:41.277334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724340+0000","last_change":"2024-03-26T04:40:55.037221+0000","last_active":"2024-03-26T04:41:17.724340+0000","last_peered":"2024-03-26T04:41:17.724340+0000","last_clean":"2024-03-26T04:41:17.724340+0000","last_became_active":"2024-03-26T04:40:55.036715+0000","last_became_peered":"2024-03-26T04:40:55.036715+0000","last_unstale":"2024-03-26T04:41:17.724340+0000","last_undegraded":"2024-03-26T04:41:17.724340+0000","last_fullsized":"2024-03-26T04:41:17.724340+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:49:33.384797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:24.037466+0000","last_change":"2024-03-26T04:40:53.019547+0000","last_active":"2024-03-26T04:41:24.037466+0000","last_peered":"2024-03-26T04:41:24.037466+0000","last_clean":"2024-03-26T04:41:24.037466+0000","last_became_active":"2024-03-26T04:40:53.018578+0000","last_became_peered":"2024-03-26T04:40:53.018578+0000","last_unstale":"2024-03-26T04:41:24.037466+0000","last_undegraded":"2024-03-26T04:41:24.037466+0000","last_fullsized":"2024-03-26T04:41:24.037466+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:04:39.258199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723838+0000","last_change":"2024-03-26T04:40:48.987326+0000","last_active":"2024-03-26T04:41:17.723838+0000","last_peered":"2024-03-26T04:41:17.723838+0000","last_clean":"2024-03-26T04:41:17.723838+0000","last_became_active":"2024-03-26T04:40:48.987028+0000","last_became_peered":"2024-03-26T04:40:48.987028+0000","last_unstale":"2024-03-26T04:41:17.723838+0000","last_undegraded":"2024-03-26T04:41:17.723838+0000","last_fullsized":"2024-03-26T04:41:17.723838+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:56:30.596093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'16","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.722729+0000","last_change":"2024-03-26T04:40:51.003997+0000","last_active":"2024-03-26T04:41:17.722729+0000","last_peered":"2024-03-26T04:41:17.722729+0000","last_clean":"2024-03-26T04:41:17.722729+0000","last_became_active":"2024-03-26T04:40:51.003720+0000","last_became_peered":"2024-03-26T04:40:51.003720+0000","last_unstale":"2024-03-26T04:41:17.722729+0000","last_undegraded":"2024-03-26T04:41:17.722729+0000","last_fullsized":"2024-03-26T04:41:17.722729+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:45:46.311728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726085+0000","last_change":"2024-03-26T04:40:55.034514+0000","last_active":"2024-03-26T04:41:17.726085+0000","last_peered":"2024-03-26T04:41:17.726085+0000","last_clean":"2024-03-26T04:41:17.726085+0000","last_became_active":"2024-03-26T04:40:55.033781+0000","last_became_peered":"2024-03-26T04:40:55.033781+0000","last_unstale":"2024-03-26T04:41:17.726085+0000","last_undegraded":"2024-03-26T04:41:17.726085+0000","last_fullsized":"2024-03-26T04:41:17.726085+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:31:36.722391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724512+0000","last_change":"2024-03-26T04:40:53.015183+0000","last_active":"2024-03-26T04:41:17.724512+0000","last_peered":"2024-03-26T04:41:17.724512+0000","last_clean":"2024-03-26T04:41:17.724512+0000","last_became_active":"2024-03-26T04:40:53.014856+0000","last_became_peered":"2024-03-26T04:40:53.014856+0000","last_unstale":"2024-03-26T04:41:17.724512+0000","last_undegraded":"2024-03-26T04:41:17.724512+0000","last_fullsized":"2024-03-26T04:41:17.724512+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:00:10.994529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724466+0000","last_change":"2024-03-26T04:40:48.989309+0000","last_active":"2024-03-26T04:41:17.724466+0000","last_peered":"2024-03-26T04:41:17.724466+0000","last_clean":"2024-03-26T04:41:17.724466+0000","last_became_active":"2024-03-26T04:40:48.988125+0000","last_became_peered":"2024-03-26T04:40:48.988125+0000","last_unstale":"2024-03-26T04:41:17.724466+0000","last_undegraded":"2024-03-26T04:41:17.724466+0000","last_fullsized":"2024-03-26T04:41:17.724466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:27:04.786889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'40","reported_seq":97,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723073+0000","last_change":"2024-03-26T04:40:51.004703+0000","last_active":"2024-03-26T04:41:17.723073+0000","last_peered":"2024-03-26T04:41:17.723073+0000","last_clean":"2024-03-26T04:41:17.723073+0000","last_became_active":"2024-03-26T04:40:51.004335+0000","last_became_peered":"2024-03-26T04:40:51.004335+0000","last_unstale":"2024-03-26T04:41:17.723073+0000","last_undegraded":"2024-03-26T04:41:17.723073+0000","last_fullsized":"2024-03-26T04:41:17.723073+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:34:42.991846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725303+0000","last_change":"2024-03-26T04:40:55.036039+0000","last_active":"2024-03-26T04:41:17.725303+0000","last_peered":"2024-03-26T04:41:17.725303+0000","last_clean":"2024-03-26T04:41:17.725303+0000","last_became_active":"2024-03-26T04:40:55.034687+0000","last_became_peered":"2024-03-26T04:40:55.034687+0000","last_unstale":"2024-03-26T04:41:17.725303+0000","last_undegraded":"2024-03-26T04:41:17.725303+0000","last_fullsized":"2024-03-26T04:41:17.725303+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:56:19.266397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.722875+0000","last_change":"2024-03-26T04:40:53.014468+0000","last_active":"2024-03-26T04:41:17.722875+0000","last_peered":"2024-03-26T04:41:17.722875+0000","last_clean":"2024-03-26T04:41:17.722875+0000","last_became_active":"2024-03-26T04:40:53.014093+0000","last_became_peered":"2024-03-26T04:40:53.014093+0000","last_unstale":"2024-03-26T04:41:17.722875+0000","last_undegraded":"2024-03-26T04:41:17.722875+0000","last_fullsized":"2024-03-26T04:41:17.722875+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:02:06.377200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727687+0000","last_change":"2024-03-26T04:40:48.986765+0000","last_active":"2024-03-26T04:41:17.727687+0000","last_peered":"2024-03-26T04:41:17.727687+0000","last_clean":"2024-03-26T04:41:17.727687+0000","last_became_active":"2024-03-26T04:40:48.985580+0000","last_became_peered":"2024-03-26T04:40:48.985580+0000","last_unstale":"2024-03-26T04:41:17.727687+0000","last_undegraded":"2024-03-26T04:41:17.727687+0000","last_fullsized":"2024-03-26T04:41:17.727687+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:36:48.989979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726901+0000","last_change":"2024-03-26T04:40:51.004031+0000","last_active":"2024-03-26T04:41:17.726901+0000","last_peered":"2024-03-26T04:41:17.726901+0000","last_clean":"2024-03-26T04:41:17.726901+0000","last_became_active":"2024-03-26T04:40:51.003511+0000","last_became_peered":"2024-03-26T04:40:51.003511+0000","last_unstale":"2024-03-26T04:41:17.726901+0000","last_undegraded":"2024-03-26T04:41:17.726901+0000","last_fullsized":"2024-03-26T04:41:17.726901+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:33:38.949456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724737+0000","last_change":"2024-03-26T04:40:55.035576+0000","last_active":"2024-03-26T04:41:17.724737+0000","last_peered":"2024-03-26T04:41:17.724737+0000","last_clean":"2024-03-26T04:41:17.724737+0000","last_became_active":"2024-03-26T04:40:55.034466+0000","last_became_peered":"2024-03-26T04:40:55.034466+0000","last_unstale":"2024-03-26T04:41:17.724737+0000","last_undegraded":"2024-03-26T04:41:17.724737+0000","last_fullsized":"2024-03-26T04:41:17.724737+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:14:07.607528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723287+0000","last_change":"2024-03-26T04:40:53.016928+0000","last_active":"2024-03-26T04:41:17.723287+0000","last_peered":"2024-03-26T04:41:17.723287+0000","last_clean":"2024-03-26T04:41:17.723287+0000","last_became_active":"2024-03-26T04:40:53.016618+0000","last_became_peered":"2024-03-26T04:40:53.016618+0000","last_unstale":"2024-03-26T04:41:17.723287+0000","last_undegraded":"2024-03-26T04:41:17.723287+0000","last_fullsized":"2024-03-26T04:41:17.723287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:45:30.482524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725964+0000","last_change":"2024-03-26T04:40:48.988419+0000","last_active":"2024-03-26T04:41:17.725964+0000","last_peered":"2024-03-26T04:41:17.725964+0000","last_clean":"2024-03-26T04:41:17.725964+0000","last_became_active":"2024-03-26T04:40:48.987898+0000","last_became_peered":"2024-03-26T04:40:48.987898+0000","last_unstale":"2024-03-26T04:41:17.725964+0000","last_undegraded":"2024-03-26T04:41:17.725964+0000","last_fullsized":"2024-03-26T04:41:17.725964+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:07:16.252180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'12","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726079+0000","last_change":"2024-03-26T04:40:51.007982+0000","last_active":"2024-03-26T04:41:17.726079+0000","last_peered":"2024-03-26T04:41:17.726079+0000","last_clean":"2024-03-26T04:41:17.726079+0000","last_became_active":"2024-03-26T04:40:51.005743+0000","last_became_peered":"2024-03-26T04:40:51.005743+0000","last_unstale":"2024-03-26T04:41:17.726079+0000","last_undegraded":"2024-03-26T04:41:17.726079+0000","last_fullsized":"2024-03-26T04:41:17.726079+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:00:53.009774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724776+0000","last_change":"2024-03-26T04:40:55.027183+0000","last_active":"2024-03-26T04:41:17.724776+0000","last_peered":"2024-03-26T04:41:17.724776+0000","last_clean":"2024-03-26T04:41:17.724776+0000","last_became_active":"2024-03-26T04:40:55.026374+0000","last_became_peered":"2024-03-26T04:40:55.026374+0000","last_unstale":"2024-03-26T04:41:17.724776+0000","last_undegraded":"2024-03-26T04:41:17.724776+0000","last_fullsized":"2024-03-26T04:41:17.724776+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:05:37.844647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726295+0000","last_change":"2024-03-26T04:40:53.015276+0000","last_active":"2024-03-26T04:41:17.726295+0000","last_peered":"2024-03-26T04:41:17.726295+0000","last_clean":"2024-03-26T04:41:17.726295+0000","last_became_active":"2024-03-26T04:40:53.014935+0000","last_became_peered":"2024-03-26T04:40:53.014935+0000","last_unstale":"2024-03-26T04:41:17.726295+0000","last_undegraded":"2024-03-26T04:41:17.726295+0000","last_fullsized":"2024-03-26T04:41:17.726295+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:44:32.939584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727465+0000","last_change":"2024-03-26T04:40:48.988350+0000","last_active":"2024-03-26T04:41:17.727465+0000","last_peered":"2024-03-26T04:41:17.727465+0000","last_clean":"2024-03-26T04:41:17.727465+0000","last_became_active":"2024-03-26T04:40:48.988043+0000","last_became_peered":"2024-03-26T04:40:48.988043+0000","last_unstale":"2024-03-26T04:41:17.727465+0000","last_undegraded":"2024-03-26T04:41:17.727465+0000","last_fullsized":"2024-03-26T04:41:17.727465+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:02:14.036599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'22","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726360+0000","last_change":"2024-03-26T04:40:51.003788+0000","last_active":"2024-03-26T04:41:17.726360+0000","last_peered":"2024-03-26T04:41:17.726360+0000","last_clean":"2024-03-26T04:41:17.726360+0000","last_became_active":"2024-03-26T04:40:51.002947+0000","last_became_peered":"2024-03-26T04:40:51.002947+0000","last_unstale":"2024-03-26T04:41:17.726360+0000","last_undegraded":"2024-03-26T04:41:17.726360+0000","last_fullsized":"2024-03-26T04:41:17.726360+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:39:21.148280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724489+0000","last_change":"2024-03-26T04:40:55.032805+0000","last_active":"2024-03-26T04:41:17.724489+0000","last_peered":"2024-03-26T04:41:17.724489+0000","last_clean":"2024-03-26T04:41:17.724489+0000","last_became_active":"2024-03-26T04:40:55.032521+0000","last_became_peered":"2024-03-26T04:40:55.032521+0000","last_unstale":"2024-03-26T04:41:17.724489+0000","last_undegraded":"2024-03-26T04:41:17.724489+0000","last_fullsized":"2024-03-26T04:41:17.724489+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:53:25.199791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723196+0000","last_change":"2024-03-26T04:40:53.015704+0000","last_active":"2024-03-26T04:41:17.723196+0000","last_peered":"2024-03-26T04:41:17.723196+0000","last_clean":"2024-03-26T04:41:17.723196+0000","last_became_active":"2024-03-26T04:40:53.014853+0000","last_became_peered":"2024-03-26T04:40:53.014853+0000","last_unstale":"2024-03-26T04:41:17.723196+0000","last_undegraded":"2024-03-26T04:41:17.723196+0000","last_fullsized":"2024-03-26T04:41:17.723196+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:00:28.898335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726012+0000","last_change":"2024-03-26T04:40:48.987118+0000","last_active":"2024-03-26T04:41:17.726012+0000","last_peered":"2024-03-26T04:41:17.726012+0000","last_clean":"2024-03-26T04:41:17.726012+0000","last_became_active":"2024-03-26T04:40:48.986403+0000","last_became_peered":"2024-03-26T04:40:48.986403+0000","last_unstale":"2024-03-26T04:41:17.726012+0000","last_undegraded":"2024-03-26T04:41:17.726012+0000","last_fullsized":"2024-03-26T04:41:17.726012+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:39:02.276252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727109+0000","last_change":"2024-03-26T04:40:51.008049+0000","last_active":"2024-03-26T04:41:17.727109+0000","last_peered":"2024-03-26T04:41:17.727109+0000","last_clean":"2024-03-26T04:41:17.727109+0000","last_became_active":"2024-03-26T04:40:51.007754+0000","last_became_peered":"2024-03-26T04:40:51.007754+0000","last_unstale":"2024-03-26T04:41:17.727109+0000","last_undegraded":"2024-03-26T04:41:17.727109+0000","last_fullsized":"2024-03-26T04:41:17.727109+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:50:26.000993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724544+0000","last_change":"2024-03-26T04:40:55.039114+0000","last_active":"2024-03-26T04:41:17.724544+0000","last_peered":"2024-03-26T04:41:17.724544+0000","last_clean":"2024-03-26T04:41:17.724544+0000","last_became_active":"2024-03-26T04:40:55.038593+0000","last_became_peered":"2024-03-26T04:40:55.038593+0000","last_unstale":"2024-03-26T04:41:17.724544+0000","last_undegraded":"2024-03-26T04:41:17.724544+0000","last_fullsized":"2024-03-26T04:41:17.724544+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:19:42.124560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:24.037430+0000","last_change":"2024-03-26T04:40:53.018678+0000","last_active":"2024-03-26T04:41:24.037430+0000","last_peered":"2024-03-26T04:41:24.037430+0000","last_clean":"2024-03-26T04:41:24.037430+0000","last_became_active":"2024-03-26T04:40:53.018179+0000","last_became_peered":"2024-03-26T04:40:53.018179+0000","last_unstale":"2024-03-26T04:41:24.037430+0000","last_undegraded":"2024-03-26T04:41:24.037430+0000","last_fullsized":"2024-03-26T04:41:24.037430+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:36:45.668419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725018+0000","last_change":"2024-03-26T04:40:48.985372+0000","last_active":"2024-03-26T04:41:17.725018+0000","last_peered":"2024-03-26T04:41:17.725018+0000","last_clean":"2024-03-26T04:41:17.725018+0000","last_became_active":"2024-03-26T04:40:48.985082+0000","last_became_peered":"2024-03-26T04:40:48.985082+0000","last_unstale":"2024-03-26T04:41:17.725018+0000","last_undegraded":"2024-03-26T04:41:17.725018+0000","last_fullsized":"2024-03-26T04:41:17.725018+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:46:43.907917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723672+0000","last_change":"2024-03-26T04:40:51.005697+0000","last_active":"2024-03-26T04:41:17.723672+0000","last_peered":"2024-03-26T04:41:17.723672+0000","last_clean":"2024-03-26T04:41:17.723672+0000","last_became_active":"2024-03-26T04:40:51.004514+0000","last_became_peered":"2024-03-26T04:40:51.004514+0000","last_unstale":"2024-03-26T04:41:17.723672+0000","last_undegraded":"2024-03-26T04:41:17.723672+0000","last_fullsized":"2024-03-26T04:41:17.723672+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:20:26.911761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725112+0000","last_change":"2024-03-26T04:40:55.033693+0000","last_active":"2024-03-26T04:41:17.725112+0000","last_peered":"2024-03-26T04:41:17.725112+0000","last_clean":"2024-03-26T04:41:17.725112+0000","last_became_active":"2024-03-26T04:40:55.033104+0000","last_became_peered":"2024-03-26T04:40:55.033104+0000","last_unstale":"2024-03-26T04:41:17.725112+0000","last_undegraded":"2024-03-26T04:41:17.725112+0000","last_fullsized":"2024-03-26T04:41:17.725112+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:48:44.321905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:24.037486+0000","last_change":"2024-03-26T04:40:53.012728+0000","last_active":"2024-03-26T04:41:24.037486+0000","last_peered":"2024-03-26T04:41:24.037486+0000","last_clean":"2024-03-26T04:41:24.037486+0000","last_became_active":"2024-03-26T04:40:53.010947+0000","last_became_peered":"2024-03-26T04:40:53.010947+0000","last_unstale":"2024-03-26T04:41:24.037486+0000","last_undegraded":"2024-03-26T04:41:24.037486+0000","last_fullsized":"2024-03-26T04:41:24.037486+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:49:44.695783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725167+0000","last_change":"2024-03-26T04:40:48.989957+0000","last_active":"2024-03-26T04:41:17.725167+0000","last_peered":"2024-03-26T04:41:17.725167+0000","last_clean":"2024-03-26T04:41:17.725167+0000","last_became_active":"2024-03-26T04:40:48.989366+0000","last_became_peered":"2024-03-26T04:40:48.989366+0000","last_unstale":"2024-03-26T04:41:17.725167+0000","last_undegraded":"2024-03-26T04:41:17.725167+0000","last_fullsized":"2024-03-26T04:41:17.725167+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:43:04.145397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726652+0000","last_change":"2024-03-26T04:40:51.003786+0000","last_active":"2024-03-26T04:41:17.726652+0000","last_peered":"2024-03-26T04:41:17.726652+0000","last_clean":"2024-03-26T04:41:17.726652+0000","last_became_active":"2024-03-26T04:40:51.003231+0000","last_became_peered":"2024-03-26T04:40:51.003231+0000","last_unstale":"2024-03-26T04:41:17.726652+0000","last_undegraded":"2024-03-26T04:41:17.726652+0000","last_fullsized":"2024-03-26T04:41:17.726652+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:11:47.699178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725307+0000","last_change":"2024-03-26T04:40:55.035942+0000","last_active":"2024-03-26T04:41:17.725307+0000","last_peered":"2024-03-26T04:41:17.725307+0000","last_clean":"2024-03-26T04:41:17.725307+0000","last_became_active":"2024-03-26T04:40:55.034699+0000","last_became_peered":"2024-03-26T04:40:55.034699+0000","last_unstale":"2024-03-26T04:41:17.725307+0000","last_undegraded":"2024-03-26T04:41:17.725307+0000","last_fullsized":"2024-03-26T04:41:17.725307+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:59:36.631121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725867+0000","last_change":"2024-03-26T04:40:53.015663+0000","last_active":"2024-03-26T04:41:17.725867+0000","last_peered":"2024-03-26T04:41:17.725867+0000","last_clean":"2024-03-26T04:41:17.725867+0000","last_became_active":"2024-03-26T04:40:53.014744+0000","last_became_peered":"2024-03-26T04:40:53.014744+0000","last_unstale":"2024-03-26T04:41:17.725867+0000","last_undegraded":"2024-03-26T04:41:17.725867+0000","last_fullsized":"2024-03-26T04:41:17.725867+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:26:56.182417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724891+0000","last_change":"2024-03-26T04:40:48.989340+0000","last_active":"2024-03-26T04:41:17.724891+0000","last_peered":"2024-03-26T04:41:17.724891+0000","last_clean":"2024-03-26T04:41:17.724891+0000","last_became_active":"2024-03-26T04:40:48.988963+0000","last_became_peered":"2024-03-26T04:40:48.988963+0000","last_unstale":"2024-03-26T04:41:17.724891+0000","last_undegraded":"2024-03-26T04:41:17.724891+0000","last_fullsized":"2024-03-26T04:41:17.724891+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:35:05.758146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724250+0000","last_change":"2024-03-26T04:40:51.005236+0000","last_active":"2024-03-26T04:41:17.724250+0000","last_peered":"2024-03-26T04:41:17.724250+0000","last_clean":"2024-03-26T04:41:17.724250+0000","last_became_active":"2024-03-26T04:40:51.004807+0000","last_became_peered":"2024-03-26T04:40:51.004807+0000","last_unstale":"2024-03-26T04:41:17.724250+0000","last_undegraded":"2024-03-26T04:41:17.724250+0000","last_fullsized":"2024-03-26T04:41:17.724250+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:08:39.529953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725622+0000","last_change":"2024-03-26T04:40:55.038784+0000","last_active":"2024-03-26T04:41:17.725622+0000","last_peered":"2024-03-26T04:41:17.725622+0000","last_clean":"2024-03-26T04:41:17.725622+0000","last_became_active":"2024-03-26T04:40:55.038414+0000","last_became_peered":"2024-03-26T04:40:55.038414+0000","last_unstale":"2024-03-26T04:41:17.725622+0000","last_undegraded":"2024-03-26T04:41:17.725622+0000","last_fullsized":"2024-03-26T04:41:17.725622+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:39:57.011678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724602+0000","last_change":"2024-03-26T04:40:53.016699+0000","last_active":"2024-03-26T04:41:17.724602+0000","last_peered":"2024-03-26T04:41:17.724602+0000","last_clean":"2024-03-26T04:41:17.724602+0000","last_became_active":"2024-03-26T04:40:53.016393+0000","last_became_peered":"2024-03-26T04:40:53.016393+0000","last_unstale":"2024-03-26T04:41:17.724602+0000","last_undegraded":"2024-03-26T04:41:17.724602+0000","last_fullsized":"2024-03-26T04:41:17.724602+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:11:10.289378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724574+0000","last_change":"2024-03-26T04:40:48.984958+0000","last_active":"2024-03-26T04:41:17.724574+0000","last_peered":"2024-03-26T04:41:17.724574+0000","last_clean":"2024-03-26T04:41:17.724574+0000","last_became_active":"2024-03-26T04:40:48.984577+0000","last_became_peered":"2024-03-26T04:40:48.984577+0000","last_unstale":"2024-03-26T04:41:17.724574+0000","last_undegraded":"2024-03-26T04:41:17.724574+0000","last_fullsized":"2024-03-26T04:41:17.724574+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:38:54.415393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725322+0000","last_change":"2024-03-26T04:40:51.009430+0000","last_active":"2024-03-26T04:41:17.725322+0000","last_peered":"2024-03-26T04:41:17.725322+0000","last_clean":"2024-03-26T04:41:17.725322+0000","last_became_active":"2024-03-26T04:40:51.009140+0000","last_became_peered":"2024-03-26T04:40:51.009140+0000","last_unstale":"2024-03-26T04:41:17.725322+0000","last_undegraded":"2024-03-26T04:41:17.725322+0000","last_fullsized":"2024-03-26T04:41:17.725322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:52:19.542590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724110+0000","last_change":"2024-03-26T04:40:55.033768+0000","last_active":"2024-03-26T04:41:17.724110+0000","last_peered":"2024-03-26T04:41:17.724110+0000","last_clean":"2024-03-26T04:41:17.724110+0000","last_became_active":"2024-03-26T04:40:55.033475+0000","last_became_peered":"2024-03-26T04:40:55.033475+0000","last_unstale":"2024-03-26T04:41:17.724110+0000","last_undegraded":"2024-03-26T04:41:17.724110+0000","last_fullsized":"2024-03-26T04:41:17.724110+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:58:01.185151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727101+0000","last_change":"2024-03-26T04:40:48.988736+0000","last_active":"2024-03-26T04:41:17.727101+0000","last_peered":"2024-03-26T04:41:17.727101+0000","last_clean":"2024-03-26T04:41:17.727101+0000","last_became_active":"2024-03-26T04:40:48.988316+0000","last_became_peered":"2024-03-26T04:40:48.988316+0000","last_unstale":"2024-03-26T04:41:17.727101+0000","last_undegraded":"2024-03-26T04:41:17.727101+0000","last_fullsized":"2024-03-26T04:41:17.727101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:43:11.713584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727211+0000","last_change":"2024-03-26T04:40:51.009233+0000","last_active":"2024-03-26T04:41:17.727211+0000","last_peered":"2024-03-26T04:41:17.727211+0000","last_clean":"2024-03-26T04:41:17.727211+0000","last_became_active":"2024-03-26T04:40:51.008966+0000","last_became_peered":"2024-03-26T04:40:51.008966+0000","last_unstale":"2024-03-26T04:41:17.727211+0000","last_undegraded":"2024-03-26T04:41:17.727211+0000","last_fullsized":"2024-03-26T04:41:17.727211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:42:06.024916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:24.037494+0000","last_change":"2024-03-26T04:40:53.013411+0000","last_active":"2024-03-26T04:41:24.037494+0000","last_peered":"2024-03-26T04:41:24.037494+0000","last_clean":"2024-03-26T04:41:24.037494+0000","last_became_active":"2024-03-26T04:40:53.012940+0000","last_became_peered":"2024-03-26T04:40:53.012940+0000","last_unstale":"2024-03-26T04:41:24.037494+0000","last_undegraded":"2024-03-26T04:41:24.037494+0000","last_fullsized":"2024-03-26T04:41:24.037494+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:36:33.280740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.722943+0000","last_change":"2024-03-26T04:40:55.037702+0000","last_active":"2024-03-26T04:41:17.722943+0000","last_peered":"2024-03-26T04:41:17.722943+0000","last_clean":"2024-03-26T04:41:17.722943+0000","last_became_active":"2024-03-26T04:40:55.037348+0000","last_became_peered":"2024-03-26T04:40:55.037348+0000","last_unstale":"2024-03-26T04:41:17.722943+0000","last_undegraded":"2024-03-26T04:41:17.722943+0000","last_fullsized":"2024-03-26T04:41:17.722943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:36:20.073037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725269+0000","last_change":"2024-03-26T04:40:48.989785+0000","last_active":"2024-03-26T04:41:17.725269+0000","last_peered":"2024-03-26T04:41:17.725269+0000","last_clean":"2024-03-26T04:41:17.725269+0000","last_became_active":"2024-03-26T04:40:48.989451+0000","last_became_peered":"2024-03-26T04:40:48.989451+0000","last_unstale":"2024-03-26T04:41:17.725269+0000","last_undegraded":"2024-03-26T04:41:17.725269+0000","last_fullsized":"2024-03-26T04:41:17.725269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:40:31.668391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:24.037395+0000","last_change":"2024-03-26T04:40:51.006465+0000","last_active":"2024-03-26T04:41:24.037395+0000","last_peered":"2024-03-26T04:41:24.037395+0000","last_clean":"2024-03-26T04:41:24.037395+0000","last_became_active":"2024-03-26T04:40:51.006054+0000","last_became_peered":"2024-03-26T04:40:51.006054+0000","last_unstale":"2024-03-26T04:41:24.037395+0000","last_undegraded":"2024-03-26T04:41:24.037395+0000","last_fullsized":"2024-03-26T04:41:24.037395+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:14:02.084076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724872+0000","last_change":"2024-03-26T04:40:53.013326+0000","last_active":"2024-03-26T04:41:17.724872+0000","last_peered":"2024-03-26T04:41:17.724872+0000","last_clean":"2024-03-26T04:41:17.724872+0000","last_became_active":"2024-03-26T04:40:53.013004+0000","last_became_peered":"2024-03-26T04:40:53.013004+0000","last_unstale":"2024-03-26T04:41:17.724872+0000","last_undegraded":"2024-03-26T04:41:17.724872+0000","last_fullsized":"2024-03-26T04:41:17.724872+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:16:46.814029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723852+0000","last_change":"2024-03-26T04:40:55.035788+0000","last_active":"2024-03-26T04:41:17.723852+0000","last_peered":"2024-03-26T04:41:17.723852+0000","last_clean":"2024-03-26T04:41:17.723852+0000","last_became_active":"2024-03-26T04:40:55.035409+0000","last_became_peered":"2024-03-26T04:40:55.035409+0000","last_unstale":"2024-03-26T04:41:17.723852+0000","last_undegraded":"2024-03-26T04:41:17.723852+0000","last_fullsized":"2024-03-26T04:41:17.723852+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:54:49.542723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726196+0000","last_change":"2024-03-26T04:40:48.986676+0000","last_active":"2024-03-26T04:41:17.726196+0000","last_peered":"2024-03-26T04:41:17.726196+0000","last_clean":"2024-03-26T04:41:17.726196+0000","last_became_active":"2024-03-26T04:40:48.985423+0000","last_became_peered":"2024-03-26T04:40:48.985423+0000","last_unstale":"2024-03-26T04:41:17.726196+0000","last_undegraded":"2024-03-26T04:41:17.726196+0000","last_fullsized":"2024-03-26T04:41:17.726196+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:06:46.267221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'25","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726996+0000","last_change":"2024-03-26T04:40:51.003899+0000","last_active":"2024-03-26T04:41:17.726996+0000","last_peered":"2024-03-26T04:41:17.726996+0000","last_clean":"2024-03-26T04:41:17.726996+0000","last_became_active":"2024-03-26T04:40:51.003430+0000","last_became_peered":"2024-03-26T04:40:51.003430+0000","last_unstale":"2024-03-26T04:41:17.726996+0000","last_undegraded":"2024-03-26T04:41:17.726996+0000","last_fullsized":"2024-03-26T04:41:17.726996+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:02:01.508969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724637+0000","last_change":"2024-03-26T04:40:53.014621+0000","last_active":"2024-03-26T04:41:17.724637+0000","last_peered":"2024-03-26T04:41:17.724637+0000","last_clean":"2024-03-26T04:41:17.724637+0000","last_became_active":"2024-03-26T04:40:53.014252+0000","last_became_peered":"2024-03-26T04:40:53.014252+0000","last_unstale":"2024-03-26T04:41:17.724637+0000","last_undegraded":"2024-03-26T04:41:17.724637+0000","last_fullsized":"2024-03-26T04:41:17.724637+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:21:59.846359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725459+0000","last_change":"2024-03-26T04:40:55.039046+0000","last_active":"2024-03-26T04:41:17.725459+0000","last_peered":"2024-03-26T04:41:17.725459+0000","last_clean":"2024-03-26T04:41:17.725459+0000","last_became_active":"2024-03-26T04:40:55.038668+0000","last_became_peered":"2024-03-26T04:40:55.038668+0000","last_unstale":"2024-03-26T04:41:17.725459+0000","last_undegraded":"2024-03-26T04:41:17.725459+0000","last_fullsized":"2024-03-26T04:41:17.725459+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:49:04.337181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'30","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724218+0000","last_change":"2024-03-26T04:40:51.007783+0000","last_active":"2024-03-26T04:41:17.724218+0000","last_peered":"2024-03-26T04:41:17.724218+0000","last_clean":"2024-03-26T04:41:17.724218+0000","last_became_active":"2024-03-26T04:40:51.007497+0000","last_became_peered":"2024-03-26T04:40:51.007497+0000","last_unstale":"2024-03-26T04:41:17.724218+0000","last_undegraded":"2024-03-26T04:41:17.724218+0000","last_fullsized":"2024-03-26T04:41:17.724218+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:08:43.565470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723652+0000","last_change":"2024-03-26T04:40:48.986461+0000","last_active":"2024-03-26T04:41:17.723652+0000","last_peered":"2024-03-26T04:41:17.723652+0000","last_clean":"2024-03-26T04:41:17.723652+0000","last_became_active":"2024-03-26T04:40:48.986129+0000","last_became_peered":"2024-03-26T04:40:48.986129+0000","last_unstale":"2024-03-26T04:41:17.723652+0000","last_undegraded":"2024-03-26T04:41:17.723652+0000","last_fullsized":"2024-03-26T04:41:17.723652+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:06:58.459963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726020+0000","last_change":"2024-03-26T04:40:53.017096+0000","last_active":"2024-03-26T04:41:17.726020+0000","last_peered":"2024-03-26T04:41:17.726020+0000","last_clean":"2024-03-26T04:41:17.726020+0000","last_became_active":"2024-03-26T04:40:53.016293+0000","last_became_peered":"2024-03-26T04:40:53.016293+0000","last_unstale":"2024-03-26T04:41:17.726020+0000","last_undegraded":"2024-03-26T04:41:17.726020+0000","last_fullsized":"2024-03-26T04:41:17.726020+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:28:26.183679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725753+0000","last_change":"2024-03-26T04:40:55.026612+0000","last_active":"2024-03-26T04:41:17.725753+0000","last_peered":"2024-03-26T04:41:17.725753+0000","last_clean":"2024-03-26T04:41:17.725753+0000","last_became_active":"2024-03-26T04:40:55.026295+0000","last_became_peered":"2024-03-26T04:40:55.026295+0000","last_unstale":"2024-03-26T04:41:17.725753+0000","last_undegraded":"2024-03-26T04:41:17.725753+0000","last_fullsized":"2024-03-26T04:41:17.725753+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:42:34.832120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723408+0000","last_change":"2024-03-26T04:40:51.005813+0000","last_active":"2024-03-26T04:41:17.723408+0000","last_peered":"2024-03-26T04:41:17.723408+0000","last_clean":"2024-03-26T04:41:17.723408+0000","last_became_active":"2024-03-26T04:40:51.004468+0000","last_became_peered":"2024-03-26T04:40:51.004468+0000","last_unstale":"2024-03-26T04:41:17.723408+0000","last_undegraded":"2024-03-26T04:41:17.723408+0000","last_fullsized":"2024-03-26T04:41:17.723408+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:49:34.563695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726513+0000","last_change":"2024-03-26T04:40:48.989398+0000","last_active":"2024-03-26T04:41:17.726513+0000","last_peered":"2024-03-26T04:41:17.726513+0000","last_clean":"2024-03-26T04:41:17.726513+0000","last_became_active":"2024-03-26T04:40:48.989076+0000","last_became_peered":"2024-03-26T04:40:48.989076+0000","last_unstale":"2024-03-26T04:41:17.726513+0000","last_undegraded":"2024-03-26T04:41:17.726513+0000","last_fullsized":"2024-03-26T04:41:17.726513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:40:54.807222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:19.824638+0000","last_change":"2024-03-26T04:40:32.292330+0000","last_active":"2024-03-26T04:41:19.824638+0000","last_peered":"2024-03-26T04:41:19.824638+0000","last_clean":"2024-03-26T04:41:19.824638+0000","last_became_active":"2024-03-26T04:40:32.291982+0000","last_became_peered":"2024-03-26T04:40:32.291982+0000","last_unstale":"2024-03-26T04:41:19.824638+0000","last_undegraded":"2024-03-26T04:41:19.824638+0000","last_fullsized":"2024-03-26T04:41:19.824638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:31.279090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:31.279090+0000","last_clean_scrub_stamp":"2024-03-26T04:40:31.279090+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:46:42.399397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725662+0000","last_change":"2024-03-26T04:40:53.015246+0000","last_active":"2024-03-26T04:41:17.725662+0000","last_peered":"2024-03-26T04:41:17.725662+0000","last_clean":"2024-03-26T04:41:17.725662+0000","last_became_active":"2024-03-26T04:40:53.013906+0000","last_became_peered":"2024-03-26T04:40:53.013906+0000","last_unstale":"2024-03-26T04:41:17.725662+0000","last_undegraded":"2024-03-26T04:41:17.725662+0000","last_fullsized":"2024-03-26T04:41:17.725662+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:12:04.575786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726518+0000","last_change":"2024-03-26T04:40:55.033985+0000","last_active":"2024-03-26T04:41:17.726518+0000","last_peered":"2024-03-26T04:41:17.726518+0000","last_clean":"2024-03-26T04:41:17.726518+0000","last_became_active":"2024-03-26T04:40:55.033010+0000","last_became_peered":"2024-03-26T04:40:55.033010+0000","last_unstale":"2024-03-26T04:41:17.726518+0000","last_undegraded":"2024-03-26T04:41:17.726518+0000","last_fullsized":"2024-03-26T04:41:17.726518+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:12:00.735642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'20","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724023+0000","last_change":"2024-03-26T04:40:51.006682+0000","last_active":"2024-03-26T04:41:17.724023+0000","last_peered":"2024-03-26T04:41:17.724023+0000","last_clean":"2024-03-26T04:41:17.724023+0000","last_became_active":"2024-03-26T04:40:51.005388+0000","last_became_peered":"2024-03-26T04:40:51.005388+0000","last_unstale":"2024-03-26T04:41:17.724023+0000","last_undegraded":"2024-03-26T04:41:17.724023+0000","last_fullsized":"2024-03-26T04:41:17.724023+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:55:30.431081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724330+0000","last_change":"2024-03-26T04:40:48.987447+0000","last_active":"2024-03-26T04:41:17.724330+0000","last_peered":"2024-03-26T04:41:17.724330+0000","last_clean":"2024-03-26T04:41:17.724330+0000","last_became_active":"2024-03-26T04:40:48.986734+0000","last_became_peered":"2024-03-26T04:40:48.986734+0000","last_unstale":"2024-03-26T04:41:17.724330+0000","last_undegraded":"2024-03-26T04:41:17.724330+0000","last_fullsized":"2024-03-26T04:41:17.724330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:24:46.655209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726668+0000","last_change":"2024-03-26T04:40:53.018269+0000","last_active":"2024-03-26T04:41:17.726668+0000","last_peered":"2024-03-26T04:41:17.726668+0000","last_clean":"2024-03-26T04:41:17.726668+0000","last_became_active":"2024-03-26T04:40:53.017931+0000","last_became_peered":"2024-03-26T04:40:53.017931+0000","last_unstale":"2024-03-26T04:41:17.726668+0000","last_undegraded":"2024-03-26T04:41:17.726668+0000","last_fullsized":"2024-03-26T04:41:17.726668+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:31:32.483989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725199+0000","last_change":"2024-03-26T04:40:55.034341+0000","last_active":"2024-03-26T04:41:17.725199+0000","last_peered":"2024-03-26T04:41:17.725199+0000","last_clean":"2024-03-26T04:41:17.725199+0000","last_became_active":"2024-03-26T04:40:55.034029+0000","last_became_peered":"2024-03-26T04:40:55.034029+0000","last_unstale":"2024-03-26T04:41:17.725199+0000","last_undegraded":"2024-03-26T04:41:17.725199+0000","last_fullsized":"2024-03-26T04:41:17.725199+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:22:04.188164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727311+0000","last_change":"2024-03-26T04:40:48.985462+0000","last_active":"2024-03-26T04:41:17.727311+0000","last_peered":"2024-03-26T04:41:17.727311+0000","last_clean":"2024-03-26T04:41:17.727311+0000","last_became_active":"2024-03-26T04:40:48.985193+0000","last_became_peered":"2024-03-26T04:40:48.985193+0000","last_unstale":"2024-03-26T04:41:17.727311+0000","last_undegraded":"2024-03-26T04:41:17.727311+0000","last_fullsized":"2024-03-26T04:41:17.727311+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:14:19.269333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'34","reported_seq":92,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:24.037601+0000","last_change":"2024-03-26T04:40:51.006391+0000","last_active":"2024-03-26T04:41:24.037601+0000","last_peered":"2024-03-26T04:41:24.037601+0000","last_clean":"2024-03-26T04:41:24.037601+0000","last_became_active":"2024-03-26T04:40:51.006034+0000","last_became_peered":"2024-03-26T04:40:51.006034+0000","last_unstale":"2024-03-26T04:41:24.037601+0000","last_undegraded":"2024-03-26T04:41:24.037601+0000","last_fullsized":"2024-03-26T04:41:24.037601+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:02:50.790457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:24.037437+0000","last_change":"2024-03-26T04:40:53.018425+0000","last_active":"2024-03-26T04:41:24.037437+0000","last_peered":"2024-03-26T04:41:24.037437+0000","last_clean":"2024-03-26T04:41:24.037437+0000","last_became_active":"2024-03-26T04:40:53.018142+0000","last_became_peered":"2024-03-26T04:40:53.018142+0000","last_unstale":"2024-03-26T04:41:24.037437+0000","last_undegraded":"2024-03-26T04:41:24.037437+0000","last_fullsized":"2024-03-26T04:41:24.037437+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:02:22.793276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724340+0000","last_change":"2024-03-26T04:40:55.033821+0000","last_active":"2024-03-26T04:41:17.724340+0000","last_peered":"2024-03-26T04:41:17.724340+0000","last_clean":"2024-03-26T04:41:17.724340+0000","last_became_active":"2024-03-26T04:40:55.033515+0000","last_became_peered":"2024-03-26T04:40:55.033515+0000","last_unstale":"2024-03-26T04:41:17.724340+0000","last_undegraded":"2024-03-26T04:41:17.724340+0000","last_fullsized":"2024-03-26T04:41:17.724340+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:47:08.857191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725272+0000","last_change":"2024-03-26T04:40:48.985955+0000","last_active":"2024-03-26T04:41:17.725272+0000","last_peered":"2024-03-26T04:41:17.725272+0000","last_clean":"2024-03-26T04:41:17.725272+0000","last_became_active":"2024-03-26T04:40:48.985466+0000","last_became_peered":"2024-03-26T04:40:48.985466+0000","last_unstale":"2024-03-26T04:41:17.725272+0000","last_undegraded":"2024-03-26T04:41:17.725272+0000","last_fullsized":"2024-03-26T04:41:17.725272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:53:26.081275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726246+0000","last_change":"2024-03-26T04:40:51.007875+0000","last_active":"2024-03-26T04:41:17.726246+0000","last_peered":"2024-03-26T04:41:17.726246+0000","last_clean":"2024-03-26T04:41:17.726246+0000","last_became_active":"2024-03-26T04:40:51.006576+0000","last_became_peered":"2024-03-26T04:40:51.006576+0000","last_unstale":"2024-03-26T04:41:17.726246+0000","last_undegraded":"2024-03-26T04:41:17.726246+0000","last_fullsized":"2024-03-26T04:41:17.726246+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:35:04.283711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:24.037326+0000","last_change":"2024-03-26T04:40:53.016891+0000","last_active":"2024-03-26T04:41:24.037326+0000","last_peered":"2024-03-26T04:41:24.037326+0000","last_clean":"2024-03-26T04:41:24.037326+0000","last_became_active":"2024-03-26T04:40:53.016603+0000","last_became_peered":"2024-03-26T04:40:53.016603+0000","last_unstale":"2024-03-26T04:41:24.037326+0000","last_undegraded":"2024-03-26T04:41:24.037326+0000","last_fullsized":"2024-03-26T04:41:24.037326+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:06:22.897370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724824+0000","last_change":"2024-03-26T04:40:55.033463+0000","last_active":"2024-03-26T04:41:17.724824+0000","last_peered":"2024-03-26T04:41:17.724824+0000","last_clean":"2024-03-26T04:41:17.724824+0000","last_became_active":"2024-03-26T04:40:55.033210+0000","last_became_peered":"2024-03-26T04:40:55.033210+0000","last_unstale":"2024-03-26T04:41:17.724824+0000","last_undegraded":"2024-03-26T04:41:17.724824+0000","last_fullsized":"2024-03-26T04:41:17.724824+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:16:17.631012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726881+0000","last_change":"2024-03-26T04:40:48.988283+0000","last_active":"2024-03-26T04:41:17.726881+0000","last_peered":"2024-03-26T04:41:17.726881+0000","last_clean":"2024-03-26T04:41:17.726881+0000","last_became_active":"2024-03-26T04:40:48.985904+0000","last_became_peered":"2024-03-26T04:40:48.985904+0000","last_unstale":"2024-03-26T04:41:17.726881+0000","last_undegraded":"2024-03-26T04:41:17.726881+0000","last_fullsized":"2024-03-26T04:41:17.726881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:24:40.555792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725081+0000","last_change":"2024-03-26T04:40:51.006111+0000","last_active":"2024-03-26T04:41:17.725081+0000","last_peered":"2024-03-26T04:41:17.725081+0000","last_clean":"2024-03-26T04:41:17.725081+0000","last_became_active":"2024-03-26T04:40:51.004395+0000","last_became_peered":"2024-03-26T04:40:51.004395+0000","last_unstale":"2024-03-26T04:41:17.725081+0000","last_undegraded":"2024-03-26T04:41:17.725081+0000","last_fullsized":"2024-03-26T04:41:17.725081+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:18:33.429560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724466+0000","last_change":"2024-03-26T04:40:53.013418+0000","last_active":"2024-03-26T04:41:17.724466+0000","last_peered":"2024-03-26T04:41:17.724466+0000","last_clean":"2024-03-26T04:41:17.724466+0000","last_became_active":"2024-03-26T04:40:53.013142+0000","last_became_peered":"2024-03-26T04:40:53.013142+0000","last_unstale":"2024-03-26T04:41:17.724466+0000","last_undegraded":"2024-03-26T04:41:17.724466+0000","last_fullsized":"2024-03-26T04:41:17.724466+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:11:12.747099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726663+0000","last_change":"2024-03-26T04:40:55.033455+0000","last_active":"2024-03-26T04:41:17.726663+0000","last_peered":"2024-03-26T04:41:17.726663+0000","last_clean":"2024-03-26T04:41:17.726663+0000","last_became_active":"2024-03-26T04:40:55.033013+0000","last_became_peered":"2024-03-26T04:40:55.033013+0000","last_unstale":"2024-03-26T04:41:17.726663+0000","last_undegraded":"2024-03-26T04:41:17.726663+0000","last_fullsized":"2024-03-26T04:41:17.726663+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:35:56.434531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:24.037284+0000","last_change":"2024-03-26T04:40:53.014667+0000","last_active":"2024-03-26T04:41:24.037284+0000","last_peered":"2024-03-26T04:41:24.037284+0000","last_clean":"2024-03-26T04:41:24.037284+0000","last_became_active":"2024-03-26T04:40:53.014278+0000","last_became_peered":"2024-03-26T04:40:53.014278+0000","last_unstale":"2024-03-26T04:41:24.037284+0000","last_undegraded":"2024-03-26T04:41:24.037284+0000","last_fullsized":"2024-03-26T04:41:24.037284+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:02:33.124733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725263+0000","last_change":"2024-03-26T04:40:48.987055+0000","last_active":"2024-03-26T04:41:17.725263+0000","last_peered":"2024-03-26T04:41:17.725263+0000","last_clean":"2024-03-26T04:41:17.725263+0000","last_became_active":"2024-03-26T04:40:48.985073+0000","last_became_peered":"2024-03-26T04:40:48.985073+0000","last_unstale":"2024-03-26T04:41:17.725263+0000","last_undegraded":"2024-03-26T04:41:17.725263+0000","last_fullsized":"2024-03-26T04:41:17.725263+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:59:19.348919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'24","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725013+0000","last_change":"2024-03-26T04:40:51.009152+0000","last_active":"2024-03-26T04:41:17.725013+0000","last_peered":"2024-03-26T04:41:17.725013+0000","last_clean":"2024-03-26T04:41:17.725013+0000","last_became_active":"2024-03-26T04:40:51.008792+0000","last_became_peered":"2024-03-26T04:40:51.008792+0000","last_unstale":"2024-03-26T04:41:17.725013+0000","last_undegraded":"2024-03-26T04:41:17.725013+0000","last_fullsized":"2024-03-26T04:41:17.725013+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:31:14.276280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726859+0000","last_change":"2024-03-26T04:40:55.027709+0000","last_active":"2024-03-26T04:41:17.726859+0000","last_peered":"2024-03-26T04:41:17.726859+0000","last_clean":"2024-03-26T04:41:17.726859+0000","last_became_active":"2024-03-26T04:40:55.027382+0000","last_became_peered":"2024-03-26T04:40:55.027382+0000","last_unstale":"2024-03-26T04:41:17.726859+0000","last_undegraded":"2024-03-26T04:41:17.726859+0000","last_fullsized":"2024-03-26T04:41:17.726859+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:33:40.657766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725928+0000","last_change":"2024-03-26T04:40:53.015658+0000","last_active":"2024-03-26T04:41:17.725928+0000","last_peered":"2024-03-26T04:41:17.725928+0000","last_clean":"2024-03-26T04:41:17.725928+0000","last_became_active":"2024-03-26T04:40:53.015194+0000","last_became_peered":"2024-03-26T04:40:53.015194+0000","last_unstale":"2024-03-26T04:41:17.725928+0000","last_undegraded":"2024-03-26T04:41:17.725928+0000","last_fullsized":"2024-03-26T04:41:17.725928+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:08:55.582534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723487+0000","last_change":"2024-03-26T04:40:48.987423+0000","last_active":"2024-03-26T04:41:17.723487+0000","last_peered":"2024-03-26T04:41:17.723487+0000","last_clean":"2024-03-26T04:41:17.723487+0000","last_became_active":"2024-03-26T04:40:48.986511+0000","last_became_peered":"2024-03-26T04:40:48.986511+0000","last_unstale":"2024-03-26T04:41:17.723487+0000","last_undegraded":"2024-03-26T04:41:17.723487+0000","last_fullsized":"2024-03-26T04:41:17.723487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:05:49.035782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'27","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724611+0000","last_change":"2024-03-26T04:40:51.007321+0000","last_active":"2024-03-26T04:41:17.724611+0000","last_peered":"2024-03-26T04:41:17.724611+0000","last_clean":"2024-03-26T04:41:17.724611+0000","last_became_active":"2024-03-26T04:40:51.007012+0000","last_became_peered":"2024-03-26T04:40:51.007012+0000","last_unstale":"2024-03-26T04:41:17.724611+0000","last_undegraded":"2024-03-26T04:41:17.724611+0000","last_fullsized":"2024-03-26T04:41:17.724611+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:22:37.763876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726660+0000","last_change":"2024-03-26T04:40:55.026964+0000","last_active":"2024-03-26T04:41:17.726660+0000","last_peered":"2024-03-26T04:41:17.726660+0000","last_clean":"2024-03-26T04:41:17.726660+0000","last_became_active":"2024-03-26T04:40:55.026499+0000","last_became_peered":"2024-03-26T04:40:55.026499+0000","last_unstale":"2024-03-26T04:41:17.726660+0000","last_undegraded":"2024-03-26T04:41:17.726660+0000","last_fullsized":"2024-03-26T04:41:17.726660+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:13:57.444687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725459+0000","last_change":"2024-03-26T04:40:48.984998+0000","last_active":"2024-03-26T04:41:17.725459+0000","last_peered":"2024-03-26T04:41:17.725459+0000","last_clean":"2024-03-26T04:41:17.725459+0000","last_became_active":"2024-03-26T04:40:48.984638+0000","last_became_peered":"2024-03-26T04:40:48.984638+0000","last_unstale":"2024-03-26T04:41:17.725459+0000","last_undegraded":"2024-03-26T04:41:17.725459+0000","last_fullsized":"2024-03-26T04:41:17.725459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:23:31.624155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723408+0000","last_change":"2024-03-26T04:40:51.005732+0000","last_active":"2024-03-26T04:41:17.723408+0000","last_peered":"2024-03-26T04:41:17.723408+0000","last_clean":"2024-03-26T04:41:17.723408+0000","last_became_active":"2024-03-26T04:40:51.004548+0000","last_became_peered":"2024-03-26T04:40:51.004548+0000","last_unstale":"2024-03-26T04:41:17.723408+0000","last_undegraded":"2024-03-26T04:41:17.723408+0000","last_fullsized":"2024-03-26T04:41:17.723408+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:25:11.762176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725416+0000","last_change":"2024-03-26T04:40:53.013931+0000","last_active":"2024-03-26T04:41:17.725416+0000","last_peered":"2024-03-26T04:41:17.725416+0000","last_clean":"2024-03-26T04:41:17.725416+0000","last_became_active":"2024-03-26T04:40:53.013508+0000","last_became_peered":"2024-03-26T04:40:53.013508+0000","last_unstale":"2024-03-26T04:41:17.725416+0000","last_undegraded":"2024-03-26T04:41:17.725416+0000","last_fullsized":"2024-03-26T04:41:17.725416+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:48:36.939752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724266+0000","last_change":"2024-03-26T04:40:55.036307+0000","last_active":"2024-03-26T04:41:17.724266+0000","last_peered":"2024-03-26T04:41:17.724266+0000","last_clean":"2024-03-26T04:41:17.724266+0000","last_became_active":"2024-03-26T04:40:55.035188+0000","last_became_peered":"2024-03-26T04:40:55.035188+0000","last_unstale":"2024-03-26T04:41:17.724266+0000","last_undegraded":"2024-03-26T04:41:17.724266+0000","last_fullsized":"2024-03-26T04:41:17.724266+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:48:28.647161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:24.037364+0000","last_change":"2024-03-26T04:40:53.015225+0000","last_active":"2024-03-26T04:41:24.037364+0000","last_peered":"2024-03-26T04:41:24.037364+0000","last_clean":"2024-03-26T04:41:24.037364+0000","last_became_active":"2024-03-26T04:40:53.014628+0000","last_became_peered":"2024-03-26T04:40:53.014628+0000","last_unstale":"2024-03-26T04:41:24.037364+0000","last_undegraded":"2024-03-26T04:41:24.037364+0000","last_fullsized":"2024-03-26T04:41:24.037364+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:53:42.384096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725316+0000","last_change":"2024-03-26T04:40:48.988583+0000","last_active":"2024-03-26T04:41:17.725316+0000","last_peered":"2024-03-26T04:41:17.725316+0000","last_clean":"2024-03-26T04:41:17.725316+0000","last_became_active":"2024-03-26T04:40:48.988186+0000","last_became_peered":"2024-03-26T04:40:48.988186+0000","last_unstale":"2024-03-26T04:41:17.725316+0000","last_undegraded":"2024-03-26T04:41:17.725316+0000","last_fullsized":"2024-03-26T04:41:17.725316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:27:24.753635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'25","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726361+0000","last_change":"2024-03-26T04:40:51.005929+0000","last_active":"2024-03-26T04:41:17.726361+0000","last_peered":"2024-03-26T04:41:17.726361+0000","last_clean":"2024-03-26T04:41:17.726361+0000","last_became_active":"2024-03-26T04:40:51.005424+0000","last_became_peered":"2024-03-26T04:40:51.005424+0000","last_unstale":"2024-03-26T04:41:17.726361+0000","last_undegraded":"2024-03-26T04:41:17.726361+0000","last_fullsized":"2024-03-26T04:41:17.726361+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:24:03.491340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724780+0000","last_change":"2024-03-26T04:40:55.027288+0000","last_active":"2024-03-26T04:41:17.724780+0000","last_peered":"2024-03-26T04:41:17.724780+0000","last_clean":"2024-03-26T04:41:17.724780+0000","last_became_active":"2024-03-26T04:40:55.027010+0000","last_became_peered":"2024-03-26T04:40:55.027010+0000","last_unstale":"2024-03-26T04:41:17.724780+0000","last_undegraded":"2024-03-26T04:41:17.724780+0000","last_fullsized":"2024-03-26T04:41:17.724780+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:09:45.859513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726335+0000","last_change":"2024-03-26T04:40:48.989060+0000","last_active":"2024-03-26T04:41:17.726335+0000","last_peered":"2024-03-26T04:41:17.726335+0000","last_clean":"2024-03-26T04:41:17.726335+0000","last_became_active":"2024-03-26T04:40:48.988487+0000","last_became_peered":"2024-03-26T04:40:48.988487+0000","last_unstale":"2024-03-26T04:41:17.726335+0000","last_undegraded":"2024-03-26T04:41:17.726335+0000","last_fullsized":"2024-03-26T04:41:17.726335+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:38:19.861083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727562+0000","last_change":"2024-03-26T04:40:51.003025+0000","last_active":"2024-03-26T04:41:17.727562+0000","last_peered":"2024-03-26T04:41:17.727562+0000","last_clean":"2024-03-26T04:41:17.727562+0000","last_became_active":"2024-03-26T04:40:51.002617+0000","last_became_peered":"2024-03-26T04:40:51.002617+0000","last_unstale":"2024-03-26T04:41:17.727562+0000","last_undegraded":"2024-03-26T04:41:17.727562+0000","last_fullsized":"2024-03-26T04:41:17.727562+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:51:57.458352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724957+0000","last_change":"2024-03-26T04:40:53.014635+0000","last_active":"2024-03-26T04:41:17.724957+0000","last_peered":"2024-03-26T04:41:17.724957+0000","last_clean":"2024-03-26T04:41:17.724957+0000","last_became_active":"2024-03-26T04:40:53.014278+0000","last_became_peered":"2024-03-26T04:40:53.014278+0000","last_unstale":"2024-03-26T04:41:17.724957+0000","last_undegraded":"2024-03-26T04:41:17.724957+0000","last_fullsized":"2024-03-26T04:41:17.724957+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:04:08.408265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.722656+0000","last_change":"2024-03-26T04:40:55.036515+0000","last_active":"2024-03-26T04:41:17.722656+0000","last_peered":"2024-03-26T04:41:17.722656+0000","last_clean":"2024-03-26T04:41:17.722656+0000","last_became_active":"2024-03-26T04:40:55.036191+0000","last_became_peered":"2024-03-26T04:40:55.036191+0000","last_unstale":"2024-03-26T04:41:17.722656+0000","last_undegraded":"2024-03-26T04:41:17.722656+0000","last_fullsized":"2024-03-26T04:41:17.722656+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:25:45.972122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727011+0000","last_change":"2024-03-26T04:40:48.988798+0000","last_active":"2024-03-26T04:41:17.727011+0000","last_peered":"2024-03-26T04:41:17.727011+0000","last_clean":"2024-03-26T04:41:17.727011+0000","last_became_active":"2024-03-26T04:40:48.988050+0000","last_became_peered":"2024-03-26T04:40:48.988050+0000","last_unstale":"2024-03-26T04:41:17.727011+0000","last_undegraded":"2024-03-26T04:41:17.727011+0000","last_fullsized":"2024-03-26T04:41:17.727011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:25:39.400286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726108+0000","last_change":"2024-03-26T04:40:51.003895+0000","last_active":"2024-03-26T04:41:17.726108+0000","last_peered":"2024-03-26T04:41:17.726108+0000","last_clean":"2024-03-26T04:41:17.726108+0000","last_became_active":"2024-03-26T04:40:51.003363+0000","last_became_peered":"2024-03-26T04:40:51.003363+0000","last_unstale":"2024-03-26T04:41:17.726108+0000","last_undegraded":"2024-03-26T04:41:17.726108+0000","last_fullsized":"2024-03-26T04:41:17.726108+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:02:03.463355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723429+0000","last_change":"2024-03-26T04:40:53.016083+0000","last_active":"2024-03-26T04:41:17.723429+0000","last_peered":"2024-03-26T04:41:17.723429+0000","last_clean":"2024-03-26T04:41:17.723429+0000","last_became_active":"2024-03-26T04:40:53.015594+0000","last_became_peered":"2024-03-26T04:40:53.015594+0000","last_unstale":"2024-03-26T04:41:17.723429+0000","last_undegraded":"2024-03-26T04:41:17.723429+0000","last_fullsized":"2024-03-26T04:41:17.723429+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:17:17.273202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724522+0000","last_change":"2024-03-26T04:40:55.033175+0000","last_active":"2024-03-26T04:41:17.724522+0000","last_peered":"2024-03-26T04:41:17.724522+0000","last_clean":"2024-03-26T04:41:17.724522+0000","last_became_active":"2024-03-26T04:40:55.032780+0000","last_became_peered":"2024-03-26T04:40:55.032780+0000","last_unstale":"2024-03-26T04:41:17.724522+0000","last_undegraded":"2024-03-26T04:41:17.724522+0000","last_fullsized":"2024-03-26T04:41:17.724522+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:41:40.707665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724891+0000","last_change":"2024-03-26T04:40:48.986308+0000","last_active":"2024-03-26T04:41:17.724891+0000","last_peered":"2024-03-26T04:41:17.724891+0000","last_clean":"2024-03-26T04:41:17.724891+0000","last_became_active":"2024-03-26T04:40:48.985995+0000","last_became_peered":"2024-03-26T04:40:48.985995+0000","last_unstale":"2024-03-26T04:41:17.724891+0000","last_undegraded":"2024-03-26T04:41:17.724891+0000","last_fullsized":"2024-03-26T04:41:17.724891+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:26:07.709967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726923+0000","last_change":"2024-03-26T04:40:51.006941+0000","last_active":"2024-03-26T04:41:17.726923+0000","last_peered":"2024-03-26T04:41:17.726923+0000","last_clean":"2024-03-26T04:41:17.726923+0000","last_became_active":"2024-03-26T04:40:51.006559+0000","last_became_peered":"2024-03-26T04:40:51.006559+0000","last_unstale":"2024-03-26T04:41:17.726923+0000","last_undegraded":"2024-03-26T04:41:17.726923+0000","last_fullsized":"2024-03-26T04:41:17.726923+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:14:56.406226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724715+0000","last_change":"2024-03-26T04:40:53.013691+0000","last_active":"2024-03-26T04:41:17.724715+0000","last_peered":"2024-03-26T04:41:17.724715+0000","last_clean":"2024-03-26T04:41:17.724715+0000","last_became_active":"2024-03-26T04:40:53.013417+0000","last_became_peered":"2024-03-26T04:40:53.013417+0000","last_unstale":"2024-03-26T04:41:17.724715+0000","last_undegraded":"2024-03-26T04:41:17.724715+0000","last_fullsized":"2024-03-26T04:41:17.724715+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:38:25.683203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":587,"ondisk_log_size":587,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411343,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738393,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:41:38.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:41:38.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:37 smithi114 ceph-mon[27627]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:41:38.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:37 smithi177 ceph-mon[27677]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:41:38.982 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:38 smithi038 ceph-mon[25836]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:39.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:38 smithi114 ceph-mon[27627]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:39.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:38 smithi177 ceph-mon[27677]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:39.283 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:39.330 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:39 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: ::ffff:172.21.15.114 - - [26/Mar/2024:04:41:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:41:41.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:40 smithi038 ceph-mon[25836]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:41.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:40 smithi114 ceph-mon[27627]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:41.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:40 smithi177 ceph-mon[27677]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:41.726 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:41 smithi038 podman[49159]: 2024-03-26 04:41:41.553467398 +0000 UTC m=+0.018542957 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T04:41:41.727 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:41 smithi038 podman[49159]: 2024-03-26 04:41:41.72655616 +0000 UTC m=+0.191631717 container create 05ae4c96274e89ad70def610be7231ad915b711299e4e30a662471f6c32b3d7e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:41.735 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:41:41.737 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-26T04:41:42.331 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:42 smithi038 podman[49159]: 2024-03-26 04:41:42.046565478 +0000 UTC m=+0.511641035 container init 05ae4c96274e89ad70def610be7231ad915b711299e4e30a662471f6c32b3d7e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:42.331 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:42 smithi038 podman[49159]: 2024-03-26 04:41:42.054518429 +0000 UTC m=+0.519593986 container start 05ae4c96274e89ad70def610be7231ad915b711299e4e30a662471f6c32b3d7e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:42.331 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[49188]: ts=2024-03-26T04:41:42.064Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T04:41:42.331 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[49188]: ts=2024-03-26T04:41:42.065Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T04:41:42.331 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[49188]: ts=2024-03-26T04:41:42.066Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.38 port=9094 2024-03-26T04:41:42.331 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[49188]: ts=2024-03-26T04:41:42.067Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T04:41:42.331 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[49188]: ts=2024-03-26T04:41:42.090Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:41:42.331 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[49188]: ts=2024-03-26T04:41:42.090Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:41:42.332 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[49188]: ts=2024-03-26T04:41:42.093Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T04:41:42.332 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[49188]: ts=2024-03-26T04:41:42.093Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T04:41:42.332 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:42 smithi038 bash[49159]: 05ae4c96274e89ad70def610be7231ad915b711299e4e30a662471f6c32b3d7e 2024-03-26T04:41:42.332 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:42 smithi038 systemd[1]: Started Ceph alertmanager.a for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:41:42.751 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:41:42] ENGINE Bus STOPPING 2024-03-26T04:41:42.751 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:41:42] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:41:42.751 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:41:42] ENGINE Bus STOPPED 2024-03-26T04:41:42.751 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:41:42] ENGINE Bus STARTING 2024-03-26T04:41:42.751 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:41:42] ENGINE Serving on http://:::9283 2024-03-26T04:41:42.751 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:42 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:41:42] ENGINE Bus STARTED 2024-03-26T04:41:42.776 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-03-26T04:41:39.738130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96612,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126556,"statfs":{"total":287972524032,"available":287873593344,"internally_reserved":0,"allocated":3158016,"data_stored":2100351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002745"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725122+0000","last_change":"2024-03-26T04:40:55.035961+0000","last_active":"2024-03-26T04:41:17.725122+0000","last_peered":"2024-03-26T04:41:17.725122+0000","last_clean":"2024-03-26T04:41:17.725122+0000","last_became_active":"2024-03-26T04:40:55.034664+0000","last_became_peered":"2024-03-26T04:40:55.034664+0000","last_unstale":"2024-03-26T04:41:17.725122+0000","last_undegraded":"2024-03-26T04:41:17.725122+0000","last_fullsized":"2024-03-26T04:41:17.725122+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:19:56.073034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724259+0000","last_change":"2024-03-26T04:40:53.013211+0000","last_active":"2024-03-26T04:41:17.724259+0000","last_peered":"2024-03-26T04:41:17.724259+0000","last_clean":"2024-03-26T04:41:17.724259+0000","last_became_active":"2024-03-26T04:40:53.011165+0000","last_became_peered":"2024-03-26T04:40:53.011165+0000","last_unstale":"2024-03-26T04:41:17.724259+0000","last_undegraded":"2024-03-26T04:41:17.724259+0000","last_fullsized":"2024-03-26T04:41:17.724259+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:16:25.123449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726192+0000","last_change":"2024-03-26T04:40:48.985731+0000","last_active":"2024-03-26T04:41:17.726192+0000","last_peered":"2024-03-26T04:41:17.726192+0000","last_clean":"2024-03-26T04:41:17.726192+0000","last_became_active":"2024-03-26T04:40:48.985418+0000","last_became_peered":"2024-03-26T04:40:48.985418+0000","last_unstale":"2024-03-26T04:41:17.726192+0000","last_undegraded":"2024-03-26T04:41:17.726192+0000","last_fullsized":"2024-03-26T04:41:17.726192+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:26:03.466720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724762+0000","last_change":"2024-03-26T04:40:51.005807+0000","last_active":"2024-03-26T04:41:17.724762+0000","last_peered":"2024-03-26T04:41:17.724762+0000","last_clean":"2024-03-26T04:41:17.724762+0000","last_became_active":"2024-03-26T04:40:51.004795+0000","last_became_peered":"2024-03-26T04:40:51.004795+0000","last_unstale":"2024-03-26T04:41:17.724762+0000","last_undegraded":"2024-03-26T04:41:17.724762+0000","last_fullsized":"2024-03-26T04:41:17.724762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:19:54.426252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723243+0000","last_change":"2024-03-26T04:40:55.035961+0000","last_active":"2024-03-26T04:41:17.723243+0000","last_peered":"2024-03-26T04:41:17.723243+0000","last_clean":"2024-03-26T04:41:17.723243+0000","last_became_active":"2024-03-26T04:40:55.035286+0000","last_became_peered":"2024-03-26T04:40:55.035286+0000","last_unstale":"2024-03-26T04:41:17.723243+0000","last_undegraded":"2024-03-26T04:41:17.723243+0000","last_fullsized":"2024-03-26T04:41:17.723243+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:25:54.077476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724811+0000","last_change":"2024-03-26T04:40:53.015612+0000","last_active":"2024-03-26T04:41:17.724811+0000","last_peered":"2024-03-26T04:41:17.724811+0000","last_clean":"2024-03-26T04:41:17.724811+0000","last_became_active":"2024-03-26T04:40:53.015267+0000","last_became_peered":"2024-03-26T04:40:53.015267+0000","last_unstale":"2024-03-26T04:41:17.724811+0000","last_undegraded":"2024-03-26T04:41:17.724811+0000","last_fullsized":"2024-03-26T04:41:17.724811+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:02:55.971241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725436+0000","last_change":"2024-03-26T04:40:48.990625+0000","last_active":"2024-03-26T04:41:17.725436+0000","last_peered":"2024-03-26T04:41:17.725436+0000","last_clean":"2024-03-26T04:41:17.725436+0000","last_became_active":"2024-03-26T04:40:48.989155+0000","last_became_peered":"2024-03-26T04:40:48.989155+0000","last_unstale":"2024-03-26T04:41:17.725436+0000","last_undegraded":"2024-03-26T04:41:17.725436+0000","last_fullsized":"2024-03-26T04:41:17.725436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:53:32.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725961+0000","last_change":"2024-03-26T04:40:51.005802+0000","last_active":"2024-03-26T04:41:17.725961+0000","last_peered":"2024-03-26T04:41:17.725961+0000","last_clean":"2024-03-26T04:41:17.725961+0000","last_became_active":"2024-03-26T04:40:51.004757+0000","last_became_peered":"2024-03-26T04:40:51.004757+0000","last_unstale":"2024-03-26T04:41:17.725961+0000","last_undegraded":"2024-03-26T04:41:17.725961+0000","last_fullsized":"2024-03-26T04:41:17.725961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:58:05.869971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724540+0000","last_change":"2024-03-26T04:40:55.032958+0000","last_active":"2024-03-26T04:41:17.724540+0000","last_peered":"2024-03-26T04:41:17.724540+0000","last_clean":"2024-03-26T04:41:17.724540+0000","last_became_active":"2024-03-26T04:40:55.032654+0000","last_became_peered":"2024-03-26T04:40:55.032654+0000","last_unstale":"2024-03-26T04:41:17.724540+0000","last_undegraded":"2024-03-26T04:41:17.724540+0000","last_fullsized":"2024-03-26T04:41:17.724540+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:44:43.531957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725690+0000","last_change":"2024-03-26T04:40:53.015200+0000","last_active":"2024-03-26T04:41:17.725690+0000","last_peered":"2024-03-26T04:41:17.725690+0000","last_clean":"2024-03-26T04:41:17.725690+0000","last_became_active":"2024-03-26T04:40:53.014211+0000","last_became_peered":"2024-03-26T04:40:53.014211+0000","last_unstale":"2024-03-26T04:41:17.725690+0000","last_undegraded":"2024-03-26T04:41:17.725690+0000","last_fullsized":"2024-03-26T04:41:17.725690+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:04:47.693417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727617+0000","last_change":"2024-03-26T04:40:48.989466+0000","last_active":"2024-03-26T04:41:17.727617+0000","last_peered":"2024-03-26T04:41:17.727617+0000","last_clean":"2024-03-26T04:41:17.727617+0000","last_became_active":"2024-03-26T04:40:48.989157+0000","last_became_peered":"2024-03-26T04:40:48.989157+0000","last_unstale":"2024-03-26T04:41:17.727617+0000","last_undegraded":"2024-03-26T04:41:17.727617+0000","last_fullsized":"2024-03-26T04:41:17.727617+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:48:47.660575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725653+0000","last_change":"2024-03-26T04:40:51.004199+0000","last_active":"2024-03-26T04:41:17.725653+0000","last_peered":"2024-03-26T04:41:17.725653+0000","last_clean":"2024-03-26T04:41:17.725653+0000","last_became_active":"2024-03-26T04:40:51.003398+0000","last_became_peered":"2024-03-26T04:40:51.003398+0000","last_unstale":"2024-03-26T04:41:17.725653+0000","last_undegraded":"2024-03-26T04:41:17.725653+0000","last_fullsized":"2024-03-26T04:41:17.725653+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:21:17.714545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725450+0000","last_change":"2024-03-26T04:40:55.034310+0000","last_active":"2024-03-26T04:41:17.725450+0000","last_peered":"2024-03-26T04:41:17.725450+0000","last_clean":"2024-03-26T04:41:17.725450+0000","last_became_active":"2024-03-26T04:40:55.033877+0000","last_became_peered":"2024-03-26T04:40:55.033877+0000","last_unstale":"2024-03-26T04:41:17.725450+0000","last_undegraded":"2024-03-26T04:41:17.725450+0000","last_fullsized":"2024-03-26T04:41:17.725450+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:59:59.353480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724729+0000","last_change":"2024-03-26T04:40:53.014572+0000","last_active":"2024-03-26T04:41:17.724729+0000","last_peered":"2024-03-26T04:41:17.724729+0000","last_clean":"2024-03-26T04:41:17.724729+0000","last_became_active":"2024-03-26T04:40:53.014176+0000","last_became_peered":"2024-03-26T04:40:53.014176+0000","last_unstale":"2024-03-26T04:41:17.724729+0000","last_undegraded":"2024-03-26T04:41:17.724729+0000","last_fullsized":"2024-03-26T04:41:17.724729+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:12:52.085798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726193+0000","last_change":"2024-03-26T04:40:48.989578+0000","last_active":"2024-03-26T04:41:17.726193+0000","last_peered":"2024-03-26T04:41:17.726193+0000","last_clean":"2024-03-26T04:41:17.726193+0000","last_became_active":"2024-03-26T04:40:48.989207+0000","last_became_peered":"2024-03-26T04:40:48.989207+0000","last_unstale":"2024-03-26T04:41:17.726193+0000","last_undegraded":"2024-03-26T04:41:17.726193+0000","last_fullsized":"2024-03-26T04:41:17.726193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:00:54.968707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'18","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727517+0000","last_change":"2024-03-26T04:40:51.008169+0000","last_active":"2024-03-26T04:41:17.727517+0000","last_peered":"2024-03-26T04:41:17.727517+0000","last_clean":"2024-03-26T04:41:17.727517+0000","last_became_active":"2024-03-26T04:40:51.007963+0000","last_became_peered":"2024-03-26T04:40:51.007963+0000","last_unstale":"2024-03-26T04:41:17.727517+0000","last_undegraded":"2024-03-26T04:41:17.727517+0000","last_fullsized":"2024-03-26T04:41:17.727517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:01:08.710764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725299+0000","last_change":"2024-03-26T04:40:55.035526+0000","last_active":"2024-03-26T04:41:17.725299+0000","last_peered":"2024-03-26T04:41:17.725299+0000","last_clean":"2024-03-26T04:41:17.725299+0000","last_became_active":"2024-03-26T04:40:55.034017+0000","last_became_peered":"2024-03-26T04:40:55.034017+0000","last_unstale":"2024-03-26T04:41:17.725299+0000","last_undegraded":"2024-03-26T04:41:17.725299+0000","last_fullsized":"2024-03-26T04:41:17.725299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:18:23.243089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726705+0000","last_change":"2024-03-26T04:40:53.018218+0000","last_active":"2024-03-26T04:41:17.726705+0000","last_peered":"2024-03-26T04:41:17.726705+0000","last_clean":"2024-03-26T04:41:17.726705+0000","last_became_active":"2024-03-26T04:40:53.017928+0000","last_became_peered":"2024-03-26T04:40:53.017928+0000","last_unstale":"2024-03-26T04:41:17.726705+0000","last_undegraded":"2024-03-26T04:41:17.726705+0000","last_fullsized":"2024-03-26T04:41:17.726705+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:30:56.831646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725528+0000","last_change":"2024-03-26T04:40:48.989361+0000","last_active":"2024-03-26T04:41:17.725528+0000","last_peered":"2024-03-26T04:41:17.725528+0000","last_clean":"2024-03-26T04:41:17.725528+0000","last_became_active":"2024-03-26T04:40:48.988913+0000","last_became_peered":"2024-03-26T04:40:48.988913+0000","last_unstale":"2024-03-26T04:41:17.725528+0000","last_undegraded":"2024-03-26T04:41:17.725528+0000","last_fullsized":"2024-03-26T04:41:17.725528+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:42:23.174794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725552+0000","last_change":"2024-03-26T04:40:51.009139+0000","last_active":"2024-03-26T04:41:17.725552+0000","last_peered":"2024-03-26T04:41:17.725552+0000","last_clean":"2024-03-26T04:41:17.725552+0000","last_became_active":"2024-03-26T04:40:51.008851+0000","last_became_peered":"2024-03-26T04:40:51.008851+0000","last_unstale":"2024-03-26T04:41:17.725552+0000","last_undegraded":"2024-03-26T04:41:17.725552+0000","last_fullsized":"2024-03-26T04:41:17.725552+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:14:51.880865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725812+0000","last_change":"2024-03-26T04:40:55.036209+0000","last_active":"2024-03-26T04:41:17.725812+0000","last_peered":"2024-03-26T04:41:17.725812+0000","last_clean":"2024-03-26T04:41:17.725812+0000","last_became_active":"2024-03-26T04:40:55.035869+0000","last_became_peered":"2024-03-26T04:40:55.035869+0000","last_unstale":"2024-03-26T04:41:17.725812+0000","last_undegraded":"2024-03-26T04:41:17.725812+0000","last_fullsized":"2024-03-26T04:41:17.725812+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:19:29.554079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726029+0000","last_change":"2024-03-26T04:40:53.012803+0000","last_active":"2024-03-26T04:41:17.726029+0000","last_peered":"2024-03-26T04:41:17.726029+0000","last_clean":"2024-03-26T04:41:17.726029+0000","last_became_active":"2024-03-26T04:40:53.011029+0000","last_became_peered":"2024-03-26T04:40:53.011029+0000","last_unstale":"2024-03-26T04:41:17.726029+0000","last_undegraded":"2024-03-26T04:41:17.726029+0000","last_fullsized":"2024-03-26T04:41:17.726029+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:40:32.531037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726012+0000","last_change":"2024-03-26T04:40:48.986540+0000","last_active":"2024-03-26T04:41:17.726012+0000","last_peered":"2024-03-26T04:41:17.726012+0000","last_clean":"2024-03-26T04:41:17.726012+0000","last_became_active":"2024-03-26T04:40:48.985152+0000","last_became_peered":"2024-03-26T04:40:48.985152+0000","last_unstale":"2024-03-26T04:41:17.726012+0000","last_undegraded":"2024-03-26T04:41:17.726012+0000","last_fullsized":"2024-03-26T04:41:17.726012+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:36:47.500789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725211+0000","last_change":"2024-03-26T04:40:51.009795+0000","last_active":"2024-03-26T04:41:17.725211+0000","last_peered":"2024-03-26T04:41:17.725211+0000","last_clean":"2024-03-26T04:41:17.725211+0000","last_became_active":"2024-03-26T04:40:51.009439+0000","last_became_peered":"2024-03-26T04:40:51.009439+0000","last_unstale":"2024-03-26T04:41:17.725211+0000","last_undegraded":"2024-03-26T04:41:17.725211+0000","last_fullsized":"2024-03-26T04:41:17.725211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:13:41.277334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724340+0000","last_change":"2024-03-26T04:40:55.037221+0000","last_active":"2024-03-26T04:41:17.724340+0000","last_peered":"2024-03-26T04:41:17.724340+0000","last_clean":"2024-03-26T04:41:17.724340+0000","last_became_active":"2024-03-26T04:40:55.036715+0000","last_became_peered":"2024-03-26T04:40:55.036715+0000","last_unstale":"2024-03-26T04:41:17.724340+0000","last_undegraded":"2024-03-26T04:41:17.724340+0000","last_fullsized":"2024-03-26T04:41:17.724340+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:49:33.384797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:29.038529+0000","last_change":"2024-03-26T04:40:53.019547+0000","last_active":"2024-03-26T04:41:29.038529+0000","last_peered":"2024-03-26T04:41:29.038529+0000","last_clean":"2024-03-26T04:41:29.038529+0000","last_became_active":"2024-03-26T04:40:53.018578+0000","last_became_peered":"2024-03-26T04:40:53.018578+0000","last_unstale":"2024-03-26T04:41:29.038529+0000","last_undegraded":"2024-03-26T04:41:29.038529+0000","last_fullsized":"2024-03-26T04:41:29.038529+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:04:39.258199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723838+0000","last_change":"2024-03-26T04:40:48.987326+0000","last_active":"2024-03-26T04:41:17.723838+0000","last_peered":"2024-03-26T04:41:17.723838+0000","last_clean":"2024-03-26T04:41:17.723838+0000","last_became_active":"2024-03-26T04:40:48.987028+0000","last_became_peered":"2024-03-26T04:40:48.987028+0000","last_unstale":"2024-03-26T04:41:17.723838+0000","last_undegraded":"2024-03-26T04:41:17.723838+0000","last_fullsized":"2024-03-26T04:41:17.723838+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:56:30.596093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'16","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.722729+0000","last_change":"2024-03-26T04:40:51.003997+0000","last_active":"2024-03-26T04:41:17.722729+0000","last_peered":"2024-03-26T04:41:17.722729+0000","last_clean":"2024-03-26T04:41:17.722729+0000","last_became_active":"2024-03-26T04:40:51.003720+0000","last_became_peered":"2024-03-26T04:40:51.003720+0000","last_unstale":"2024-03-26T04:41:17.722729+0000","last_undegraded":"2024-03-26T04:41:17.722729+0000","last_fullsized":"2024-03-26T04:41:17.722729+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:45:46.311728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726085+0000","last_change":"2024-03-26T04:40:55.034514+0000","last_active":"2024-03-26T04:41:17.726085+0000","last_peered":"2024-03-26T04:41:17.726085+0000","last_clean":"2024-03-26T04:41:17.726085+0000","last_became_active":"2024-03-26T04:40:55.033781+0000","last_became_peered":"2024-03-26T04:40:55.033781+0000","last_unstale":"2024-03-26T04:41:17.726085+0000","last_undegraded":"2024-03-26T04:41:17.726085+0000","last_fullsized":"2024-03-26T04:41:17.726085+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:31:36.722391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724512+0000","last_change":"2024-03-26T04:40:53.015183+0000","last_active":"2024-03-26T04:41:17.724512+0000","last_peered":"2024-03-26T04:41:17.724512+0000","last_clean":"2024-03-26T04:41:17.724512+0000","last_became_active":"2024-03-26T04:40:53.014856+0000","last_became_peered":"2024-03-26T04:40:53.014856+0000","last_unstale":"2024-03-26T04:41:17.724512+0000","last_undegraded":"2024-03-26T04:41:17.724512+0000","last_fullsized":"2024-03-26T04:41:17.724512+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:00:10.994529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724466+0000","last_change":"2024-03-26T04:40:48.989309+0000","last_active":"2024-03-26T04:41:17.724466+0000","last_peered":"2024-03-26T04:41:17.724466+0000","last_clean":"2024-03-26T04:41:17.724466+0000","last_became_active":"2024-03-26T04:40:48.988125+0000","last_became_peered":"2024-03-26T04:40:48.988125+0000","last_unstale":"2024-03-26T04:41:17.724466+0000","last_undegraded":"2024-03-26T04:41:17.724466+0000","last_fullsized":"2024-03-26T04:41:17.724466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:27:04.786889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'40","reported_seq":97,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723073+0000","last_change":"2024-03-26T04:40:51.004703+0000","last_active":"2024-03-26T04:41:17.723073+0000","last_peered":"2024-03-26T04:41:17.723073+0000","last_clean":"2024-03-26T04:41:17.723073+0000","last_became_active":"2024-03-26T04:40:51.004335+0000","last_became_peered":"2024-03-26T04:40:51.004335+0000","last_unstale":"2024-03-26T04:41:17.723073+0000","last_undegraded":"2024-03-26T04:41:17.723073+0000","last_fullsized":"2024-03-26T04:41:17.723073+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:34:42.991846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725303+0000","last_change":"2024-03-26T04:40:55.036039+0000","last_active":"2024-03-26T04:41:17.725303+0000","last_peered":"2024-03-26T04:41:17.725303+0000","last_clean":"2024-03-26T04:41:17.725303+0000","last_became_active":"2024-03-26T04:40:55.034687+0000","last_became_peered":"2024-03-26T04:40:55.034687+0000","last_unstale":"2024-03-26T04:41:17.725303+0000","last_undegraded":"2024-03-26T04:41:17.725303+0000","last_fullsized":"2024-03-26T04:41:17.725303+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:56:19.266397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.722875+0000","last_change":"2024-03-26T04:40:53.014468+0000","last_active":"2024-03-26T04:41:17.722875+0000","last_peered":"2024-03-26T04:41:17.722875+0000","last_clean":"2024-03-26T04:41:17.722875+0000","last_became_active":"2024-03-26T04:40:53.014093+0000","last_became_peered":"2024-03-26T04:40:53.014093+0000","last_unstale":"2024-03-26T04:41:17.722875+0000","last_undegraded":"2024-03-26T04:41:17.722875+0000","last_fullsized":"2024-03-26T04:41:17.722875+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:02:06.377200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727687+0000","last_change":"2024-03-26T04:40:48.986765+0000","last_active":"2024-03-26T04:41:17.727687+0000","last_peered":"2024-03-26T04:41:17.727687+0000","last_clean":"2024-03-26T04:41:17.727687+0000","last_became_active":"2024-03-26T04:40:48.985580+0000","last_became_peered":"2024-03-26T04:40:48.985580+0000","last_unstale":"2024-03-26T04:41:17.727687+0000","last_undegraded":"2024-03-26T04:41:17.727687+0000","last_fullsized":"2024-03-26T04:41:17.727687+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:36:48.989979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726901+0000","last_change":"2024-03-26T04:40:51.004031+0000","last_active":"2024-03-26T04:41:17.726901+0000","last_peered":"2024-03-26T04:41:17.726901+0000","last_clean":"2024-03-26T04:41:17.726901+0000","last_became_active":"2024-03-26T04:40:51.003511+0000","last_became_peered":"2024-03-26T04:40:51.003511+0000","last_unstale":"2024-03-26T04:41:17.726901+0000","last_undegraded":"2024-03-26T04:41:17.726901+0000","last_fullsized":"2024-03-26T04:41:17.726901+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:33:38.949456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724737+0000","last_change":"2024-03-26T04:40:55.035576+0000","last_active":"2024-03-26T04:41:17.724737+0000","last_peered":"2024-03-26T04:41:17.724737+0000","last_clean":"2024-03-26T04:41:17.724737+0000","last_became_active":"2024-03-26T04:40:55.034466+0000","last_became_peered":"2024-03-26T04:40:55.034466+0000","last_unstale":"2024-03-26T04:41:17.724737+0000","last_undegraded":"2024-03-26T04:41:17.724737+0000","last_fullsized":"2024-03-26T04:41:17.724737+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:14:07.607528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723287+0000","last_change":"2024-03-26T04:40:53.016928+0000","last_active":"2024-03-26T04:41:17.723287+0000","last_peered":"2024-03-26T04:41:17.723287+0000","last_clean":"2024-03-26T04:41:17.723287+0000","last_became_active":"2024-03-26T04:40:53.016618+0000","last_became_peered":"2024-03-26T04:40:53.016618+0000","last_unstale":"2024-03-26T04:41:17.723287+0000","last_undegraded":"2024-03-26T04:41:17.723287+0000","last_fullsized":"2024-03-26T04:41:17.723287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:45:30.482524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725964+0000","last_change":"2024-03-26T04:40:48.988419+0000","last_active":"2024-03-26T04:41:17.725964+0000","last_peered":"2024-03-26T04:41:17.725964+0000","last_clean":"2024-03-26T04:41:17.725964+0000","last_became_active":"2024-03-26T04:40:48.987898+0000","last_became_peered":"2024-03-26T04:40:48.987898+0000","last_unstale":"2024-03-26T04:41:17.725964+0000","last_undegraded":"2024-03-26T04:41:17.725964+0000","last_fullsized":"2024-03-26T04:41:17.725964+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:07:16.252180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'12","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726079+0000","last_change":"2024-03-26T04:40:51.007982+0000","last_active":"2024-03-26T04:41:17.726079+0000","last_peered":"2024-03-26T04:41:17.726079+0000","last_clean":"2024-03-26T04:41:17.726079+0000","last_became_active":"2024-03-26T04:40:51.005743+0000","last_became_peered":"2024-03-26T04:40:51.005743+0000","last_unstale":"2024-03-26T04:41:17.726079+0000","last_undegraded":"2024-03-26T04:41:17.726079+0000","last_fullsized":"2024-03-26T04:41:17.726079+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:00:53.009774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724776+0000","last_change":"2024-03-26T04:40:55.027183+0000","last_active":"2024-03-26T04:41:17.724776+0000","last_peered":"2024-03-26T04:41:17.724776+0000","last_clean":"2024-03-26T04:41:17.724776+0000","last_became_active":"2024-03-26T04:40:55.026374+0000","last_became_peered":"2024-03-26T04:40:55.026374+0000","last_unstale":"2024-03-26T04:41:17.724776+0000","last_undegraded":"2024-03-26T04:41:17.724776+0000","last_fullsized":"2024-03-26T04:41:17.724776+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:05:37.844647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726295+0000","last_change":"2024-03-26T04:40:53.015276+0000","last_active":"2024-03-26T04:41:17.726295+0000","last_peered":"2024-03-26T04:41:17.726295+0000","last_clean":"2024-03-26T04:41:17.726295+0000","last_became_active":"2024-03-26T04:40:53.014935+0000","last_became_peered":"2024-03-26T04:40:53.014935+0000","last_unstale":"2024-03-26T04:41:17.726295+0000","last_undegraded":"2024-03-26T04:41:17.726295+0000","last_fullsized":"2024-03-26T04:41:17.726295+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:44:32.939584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727465+0000","last_change":"2024-03-26T04:40:48.988350+0000","last_active":"2024-03-26T04:41:17.727465+0000","last_peered":"2024-03-26T04:41:17.727465+0000","last_clean":"2024-03-26T04:41:17.727465+0000","last_became_active":"2024-03-26T04:40:48.988043+0000","last_became_peered":"2024-03-26T04:40:48.988043+0000","last_unstale":"2024-03-26T04:41:17.727465+0000","last_undegraded":"2024-03-26T04:41:17.727465+0000","last_fullsized":"2024-03-26T04:41:17.727465+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:02:14.036599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'22","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726360+0000","last_change":"2024-03-26T04:40:51.003788+0000","last_active":"2024-03-26T04:41:17.726360+0000","last_peered":"2024-03-26T04:41:17.726360+0000","last_clean":"2024-03-26T04:41:17.726360+0000","last_became_active":"2024-03-26T04:40:51.002947+0000","last_became_peered":"2024-03-26T04:40:51.002947+0000","last_unstale":"2024-03-26T04:41:17.726360+0000","last_undegraded":"2024-03-26T04:41:17.726360+0000","last_fullsized":"2024-03-26T04:41:17.726360+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:39:21.148280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724489+0000","last_change":"2024-03-26T04:40:55.032805+0000","last_active":"2024-03-26T04:41:17.724489+0000","last_peered":"2024-03-26T04:41:17.724489+0000","last_clean":"2024-03-26T04:41:17.724489+0000","last_became_active":"2024-03-26T04:40:55.032521+0000","last_became_peered":"2024-03-26T04:40:55.032521+0000","last_unstale":"2024-03-26T04:41:17.724489+0000","last_undegraded":"2024-03-26T04:41:17.724489+0000","last_fullsized":"2024-03-26T04:41:17.724489+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:53:25.199791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723196+0000","last_change":"2024-03-26T04:40:53.015704+0000","last_active":"2024-03-26T04:41:17.723196+0000","last_peered":"2024-03-26T04:41:17.723196+0000","last_clean":"2024-03-26T04:41:17.723196+0000","last_became_active":"2024-03-26T04:40:53.014853+0000","last_became_peered":"2024-03-26T04:40:53.014853+0000","last_unstale":"2024-03-26T04:41:17.723196+0000","last_undegraded":"2024-03-26T04:41:17.723196+0000","last_fullsized":"2024-03-26T04:41:17.723196+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:00:28.898335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726012+0000","last_change":"2024-03-26T04:40:48.987118+0000","last_active":"2024-03-26T04:41:17.726012+0000","last_peered":"2024-03-26T04:41:17.726012+0000","last_clean":"2024-03-26T04:41:17.726012+0000","last_became_active":"2024-03-26T04:40:48.986403+0000","last_became_peered":"2024-03-26T04:40:48.986403+0000","last_unstale":"2024-03-26T04:41:17.726012+0000","last_undegraded":"2024-03-26T04:41:17.726012+0000","last_fullsized":"2024-03-26T04:41:17.726012+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:39:02.276252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727109+0000","last_change":"2024-03-26T04:40:51.008049+0000","last_active":"2024-03-26T04:41:17.727109+0000","last_peered":"2024-03-26T04:41:17.727109+0000","last_clean":"2024-03-26T04:41:17.727109+0000","last_became_active":"2024-03-26T04:40:51.007754+0000","last_became_peered":"2024-03-26T04:40:51.007754+0000","last_unstale":"2024-03-26T04:41:17.727109+0000","last_undegraded":"2024-03-26T04:41:17.727109+0000","last_fullsized":"2024-03-26T04:41:17.727109+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:50:26.000993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724544+0000","last_change":"2024-03-26T04:40:55.039114+0000","last_active":"2024-03-26T04:41:17.724544+0000","last_peered":"2024-03-26T04:41:17.724544+0000","last_clean":"2024-03-26T04:41:17.724544+0000","last_became_active":"2024-03-26T04:40:55.038593+0000","last_became_peered":"2024-03-26T04:40:55.038593+0000","last_unstale":"2024-03-26T04:41:17.724544+0000","last_undegraded":"2024-03-26T04:41:17.724544+0000","last_fullsized":"2024-03-26T04:41:17.724544+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:19:42.124560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:29.038489+0000","last_change":"2024-03-26T04:40:53.018678+0000","last_active":"2024-03-26T04:41:29.038489+0000","last_peered":"2024-03-26T04:41:29.038489+0000","last_clean":"2024-03-26T04:41:29.038489+0000","last_became_active":"2024-03-26T04:40:53.018179+0000","last_became_peered":"2024-03-26T04:40:53.018179+0000","last_unstale":"2024-03-26T04:41:29.038489+0000","last_undegraded":"2024-03-26T04:41:29.038489+0000","last_fullsized":"2024-03-26T04:41:29.038489+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:36:45.668419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725018+0000","last_change":"2024-03-26T04:40:48.985372+0000","last_active":"2024-03-26T04:41:17.725018+0000","last_peered":"2024-03-26T04:41:17.725018+0000","last_clean":"2024-03-26T04:41:17.725018+0000","last_became_active":"2024-03-26T04:40:48.985082+0000","last_became_peered":"2024-03-26T04:40:48.985082+0000","last_unstale":"2024-03-26T04:41:17.725018+0000","last_undegraded":"2024-03-26T04:41:17.725018+0000","last_fullsized":"2024-03-26T04:41:17.725018+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:46:43.907917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723672+0000","last_change":"2024-03-26T04:40:51.005697+0000","last_active":"2024-03-26T04:41:17.723672+0000","last_peered":"2024-03-26T04:41:17.723672+0000","last_clean":"2024-03-26T04:41:17.723672+0000","last_became_active":"2024-03-26T04:40:51.004514+0000","last_became_peered":"2024-03-26T04:40:51.004514+0000","last_unstale":"2024-03-26T04:41:17.723672+0000","last_undegraded":"2024-03-26T04:41:17.723672+0000","last_fullsized":"2024-03-26T04:41:17.723672+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:20:26.911761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725112+0000","last_change":"2024-03-26T04:40:55.033693+0000","last_active":"2024-03-26T04:41:17.725112+0000","last_peered":"2024-03-26T04:41:17.725112+0000","last_clean":"2024-03-26T04:41:17.725112+0000","last_became_active":"2024-03-26T04:40:55.033104+0000","last_became_peered":"2024-03-26T04:40:55.033104+0000","last_unstale":"2024-03-26T04:41:17.725112+0000","last_undegraded":"2024-03-26T04:41:17.725112+0000","last_fullsized":"2024-03-26T04:41:17.725112+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:48:44.321905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:29.038504+0000","last_change":"2024-03-26T04:40:53.012728+0000","last_active":"2024-03-26T04:41:29.038504+0000","last_peered":"2024-03-26T04:41:29.038504+0000","last_clean":"2024-03-26T04:41:29.038504+0000","last_became_active":"2024-03-26T04:40:53.010947+0000","last_became_peered":"2024-03-26T04:40:53.010947+0000","last_unstale":"2024-03-26T04:41:29.038504+0000","last_undegraded":"2024-03-26T04:41:29.038504+0000","last_fullsized":"2024-03-26T04:41:29.038504+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:49:44.695783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725167+0000","last_change":"2024-03-26T04:40:48.989957+0000","last_active":"2024-03-26T04:41:17.725167+0000","last_peered":"2024-03-26T04:41:17.725167+0000","last_clean":"2024-03-26T04:41:17.725167+0000","last_became_active":"2024-03-26T04:40:48.989366+0000","last_became_peered":"2024-03-26T04:40:48.989366+0000","last_unstale":"2024-03-26T04:41:17.725167+0000","last_undegraded":"2024-03-26T04:41:17.725167+0000","last_fullsized":"2024-03-26T04:41:17.725167+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:43:04.145397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726652+0000","last_change":"2024-03-26T04:40:51.003786+0000","last_active":"2024-03-26T04:41:17.726652+0000","last_peered":"2024-03-26T04:41:17.726652+0000","last_clean":"2024-03-26T04:41:17.726652+0000","last_became_active":"2024-03-26T04:40:51.003231+0000","last_became_peered":"2024-03-26T04:40:51.003231+0000","last_unstale":"2024-03-26T04:41:17.726652+0000","last_undegraded":"2024-03-26T04:41:17.726652+0000","last_fullsized":"2024-03-26T04:41:17.726652+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:11:47.699178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725307+0000","last_change":"2024-03-26T04:40:55.035942+0000","last_active":"2024-03-26T04:41:17.725307+0000","last_peered":"2024-03-26T04:41:17.725307+0000","last_clean":"2024-03-26T04:41:17.725307+0000","last_became_active":"2024-03-26T04:40:55.034699+0000","last_became_peered":"2024-03-26T04:40:55.034699+0000","last_unstale":"2024-03-26T04:41:17.725307+0000","last_undegraded":"2024-03-26T04:41:17.725307+0000","last_fullsized":"2024-03-26T04:41:17.725307+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:59:36.631121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725867+0000","last_change":"2024-03-26T04:40:53.015663+0000","last_active":"2024-03-26T04:41:17.725867+0000","last_peered":"2024-03-26T04:41:17.725867+0000","last_clean":"2024-03-26T04:41:17.725867+0000","last_became_active":"2024-03-26T04:40:53.014744+0000","last_became_peered":"2024-03-26T04:40:53.014744+0000","last_unstale":"2024-03-26T04:41:17.725867+0000","last_undegraded":"2024-03-26T04:41:17.725867+0000","last_fullsized":"2024-03-26T04:41:17.725867+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:26:56.182417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724891+0000","last_change":"2024-03-26T04:40:48.989340+0000","last_active":"2024-03-26T04:41:17.724891+0000","last_peered":"2024-03-26T04:41:17.724891+0000","last_clean":"2024-03-26T04:41:17.724891+0000","last_became_active":"2024-03-26T04:40:48.988963+0000","last_became_peered":"2024-03-26T04:40:48.988963+0000","last_unstale":"2024-03-26T04:41:17.724891+0000","last_undegraded":"2024-03-26T04:41:17.724891+0000","last_fullsized":"2024-03-26T04:41:17.724891+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:35:05.758146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724250+0000","last_change":"2024-03-26T04:40:51.005236+0000","last_active":"2024-03-26T04:41:17.724250+0000","last_peered":"2024-03-26T04:41:17.724250+0000","last_clean":"2024-03-26T04:41:17.724250+0000","last_became_active":"2024-03-26T04:40:51.004807+0000","last_became_peered":"2024-03-26T04:40:51.004807+0000","last_unstale":"2024-03-26T04:41:17.724250+0000","last_undegraded":"2024-03-26T04:41:17.724250+0000","last_fullsized":"2024-03-26T04:41:17.724250+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:08:39.529953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725622+0000","last_change":"2024-03-26T04:40:55.038784+0000","last_active":"2024-03-26T04:41:17.725622+0000","last_peered":"2024-03-26T04:41:17.725622+0000","last_clean":"2024-03-26T04:41:17.725622+0000","last_became_active":"2024-03-26T04:40:55.038414+0000","last_became_peered":"2024-03-26T04:40:55.038414+0000","last_unstale":"2024-03-26T04:41:17.725622+0000","last_undegraded":"2024-03-26T04:41:17.725622+0000","last_fullsized":"2024-03-26T04:41:17.725622+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:39:57.011678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724602+0000","last_change":"2024-03-26T04:40:53.016699+0000","last_active":"2024-03-26T04:41:17.724602+0000","last_peered":"2024-03-26T04:41:17.724602+0000","last_clean":"2024-03-26T04:41:17.724602+0000","last_became_active":"2024-03-26T04:40:53.016393+0000","last_became_peered":"2024-03-26T04:40:53.016393+0000","last_unstale":"2024-03-26T04:41:17.724602+0000","last_undegraded":"2024-03-26T04:41:17.724602+0000","last_fullsized":"2024-03-26T04:41:17.724602+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:11:10.289378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724574+0000","last_change":"2024-03-26T04:40:48.984958+0000","last_active":"2024-03-26T04:41:17.724574+0000","last_peered":"2024-03-26T04:41:17.724574+0000","last_clean":"2024-03-26T04:41:17.724574+0000","last_became_active":"2024-03-26T04:40:48.984577+0000","last_became_peered":"2024-03-26T04:40:48.984577+0000","last_unstale":"2024-03-26T04:41:17.724574+0000","last_undegraded":"2024-03-26T04:41:17.724574+0000","last_fullsized":"2024-03-26T04:41:17.724574+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:38:54.415393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725322+0000","last_change":"2024-03-26T04:40:51.009430+0000","last_active":"2024-03-26T04:41:17.725322+0000","last_peered":"2024-03-26T04:41:17.725322+0000","last_clean":"2024-03-26T04:41:17.725322+0000","last_became_active":"2024-03-26T04:40:51.009140+0000","last_became_peered":"2024-03-26T04:40:51.009140+0000","last_unstale":"2024-03-26T04:41:17.725322+0000","last_undegraded":"2024-03-26T04:41:17.725322+0000","last_fullsized":"2024-03-26T04:41:17.725322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:52:19.542590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724110+0000","last_change":"2024-03-26T04:40:55.033768+0000","last_active":"2024-03-26T04:41:17.724110+0000","last_peered":"2024-03-26T04:41:17.724110+0000","last_clean":"2024-03-26T04:41:17.724110+0000","last_became_active":"2024-03-26T04:40:55.033475+0000","last_became_peered":"2024-03-26T04:40:55.033475+0000","last_unstale":"2024-03-26T04:41:17.724110+0000","last_undegraded":"2024-03-26T04:41:17.724110+0000","last_fullsized":"2024-03-26T04:41:17.724110+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:58:01.185151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727101+0000","last_change":"2024-03-26T04:40:48.988736+0000","last_active":"2024-03-26T04:41:17.727101+0000","last_peered":"2024-03-26T04:41:17.727101+0000","last_clean":"2024-03-26T04:41:17.727101+0000","last_became_active":"2024-03-26T04:40:48.988316+0000","last_became_peered":"2024-03-26T04:40:48.988316+0000","last_unstale":"2024-03-26T04:41:17.727101+0000","last_undegraded":"2024-03-26T04:41:17.727101+0000","last_fullsized":"2024-03-26T04:41:17.727101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:43:11.713584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727211+0000","last_change":"2024-03-26T04:40:51.009233+0000","last_active":"2024-03-26T04:41:17.727211+0000","last_peered":"2024-03-26T04:41:17.727211+0000","last_clean":"2024-03-26T04:41:17.727211+0000","last_became_active":"2024-03-26T04:40:51.008966+0000","last_became_peered":"2024-03-26T04:40:51.008966+0000","last_unstale":"2024-03-26T04:41:17.727211+0000","last_undegraded":"2024-03-26T04:41:17.727211+0000","last_fullsized":"2024-03-26T04:41:17.727211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:42:06.024916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:29.038590+0000","last_change":"2024-03-26T04:40:53.013411+0000","last_active":"2024-03-26T04:41:29.038590+0000","last_peered":"2024-03-26T04:41:29.038590+0000","last_clean":"2024-03-26T04:41:29.038590+0000","last_became_active":"2024-03-26T04:40:53.012940+0000","last_became_peered":"2024-03-26T04:40:53.012940+0000","last_unstale":"2024-03-26T04:41:29.038590+0000","last_undegraded":"2024-03-26T04:41:29.038590+0000","last_fullsized":"2024-03-26T04:41:29.038590+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:36:33.280740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.722943+0000","last_change":"2024-03-26T04:40:55.037702+0000","last_active":"2024-03-26T04:41:17.722943+0000","last_peered":"2024-03-26T04:41:17.722943+0000","last_clean":"2024-03-26T04:41:17.722943+0000","last_became_active":"2024-03-26T04:40:55.037348+0000","last_became_peered":"2024-03-26T04:40:55.037348+0000","last_unstale":"2024-03-26T04:41:17.722943+0000","last_undegraded":"2024-03-26T04:41:17.722943+0000","last_fullsized":"2024-03-26T04:41:17.722943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:36:20.073037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725269+0000","last_change":"2024-03-26T04:40:48.989785+0000","last_active":"2024-03-26T04:41:17.725269+0000","last_peered":"2024-03-26T04:41:17.725269+0000","last_clean":"2024-03-26T04:41:17.725269+0000","last_became_active":"2024-03-26T04:40:48.989451+0000","last_became_peered":"2024-03-26T04:40:48.989451+0000","last_unstale":"2024-03-26T04:41:17.725269+0000","last_undegraded":"2024-03-26T04:41:17.725269+0000","last_fullsized":"2024-03-26T04:41:17.725269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:40:31.668391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:29.038499+0000","last_change":"2024-03-26T04:40:51.006465+0000","last_active":"2024-03-26T04:41:29.038499+0000","last_peered":"2024-03-26T04:41:29.038499+0000","last_clean":"2024-03-26T04:41:29.038499+0000","last_became_active":"2024-03-26T04:40:51.006054+0000","last_became_peered":"2024-03-26T04:40:51.006054+0000","last_unstale":"2024-03-26T04:41:29.038499+0000","last_undegraded":"2024-03-26T04:41:29.038499+0000","last_fullsized":"2024-03-26T04:41:29.038499+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:14:02.084076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724872+0000","last_change":"2024-03-26T04:40:53.013326+0000","last_active":"2024-03-26T04:41:17.724872+0000","last_peered":"2024-03-26T04:41:17.724872+0000","last_clean":"2024-03-26T04:41:17.724872+0000","last_became_active":"2024-03-26T04:40:53.013004+0000","last_became_peered":"2024-03-26T04:40:53.013004+0000","last_unstale":"2024-03-26T04:41:17.724872+0000","last_undegraded":"2024-03-26T04:41:17.724872+0000","last_fullsized":"2024-03-26T04:41:17.724872+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:16:46.814029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723852+0000","last_change":"2024-03-26T04:40:55.035788+0000","last_active":"2024-03-26T04:41:17.723852+0000","last_peered":"2024-03-26T04:41:17.723852+0000","last_clean":"2024-03-26T04:41:17.723852+0000","last_became_active":"2024-03-26T04:40:55.035409+0000","last_became_peered":"2024-03-26T04:40:55.035409+0000","last_unstale":"2024-03-26T04:41:17.723852+0000","last_undegraded":"2024-03-26T04:41:17.723852+0000","last_fullsized":"2024-03-26T04:41:17.723852+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:54:49.542723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726196+0000","last_change":"2024-03-26T04:40:48.986676+0000","last_active":"2024-03-26T04:41:17.726196+0000","last_peered":"2024-03-26T04:41:17.726196+0000","last_clean":"2024-03-26T04:41:17.726196+0000","last_became_active":"2024-03-26T04:40:48.985423+0000","last_became_peered":"2024-03-26T04:40:48.985423+0000","last_unstale":"2024-03-26T04:41:17.726196+0000","last_undegraded":"2024-03-26T04:41:17.726196+0000","last_fullsized":"2024-03-26T04:41:17.726196+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:06:46.267221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'25","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726996+0000","last_change":"2024-03-26T04:40:51.003899+0000","last_active":"2024-03-26T04:41:17.726996+0000","last_peered":"2024-03-26T04:41:17.726996+0000","last_clean":"2024-03-26T04:41:17.726996+0000","last_became_active":"2024-03-26T04:40:51.003430+0000","last_became_peered":"2024-03-26T04:40:51.003430+0000","last_unstale":"2024-03-26T04:41:17.726996+0000","last_undegraded":"2024-03-26T04:41:17.726996+0000","last_fullsized":"2024-03-26T04:41:17.726996+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:02:01.508969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724637+0000","last_change":"2024-03-26T04:40:53.014621+0000","last_active":"2024-03-26T04:41:17.724637+0000","last_peered":"2024-03-26T04:41:17.724637+0000","last_clean":"2024-03-26T04:41:17.724637+0000","last_became_active":"2024-03-26T04:40:53.014252+0000","last_became_peered":"2024-03-26T04:40:53.014252+0000","last_unstale":"2024-03-26T04:41:17.724637+0000","last_undegraded":"2024-03-26T04:41:17.724637+0000","last_fullsized":"2024-03-26T04:41:17.724637+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:21:59.846359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725459+0000","last_change":"2024-03-26T04:40:55.039046+0000","last_active":"2024-03-26T04:41:17.725459+0000","last_peered":"2024-03-26T04:41:17.725459+0000","last_clean":"2024-03-26T04:41:17.725459+0000","last_became_active":"2024-03-26T04:40:55.038668+0000","last_became_peered":"2024-03-26T04:40:55.038668+0000","last_unstale":"2024-03-26T04:41:17.725459+0000","last_undegraded":"2024-03-26T04:41:17.725459+0000","last_fullsized":"2024-03-26T04:41:17.725459+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:49:04.337181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'30","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724218+0000","last_change":"2024-03-26T04:40:51.007783+0000","last_active":"2024-03-26T04:41:17.724218+0000","last_peered":"2024-03-26T04:41:17.724218+0000","last_clean":"2024-03-26T04:41:17.724218+0000","last_became_active":"2024-03-26T04:40:51.007497+0000","last_became_peered":"2024-03-26T04:40:51.007497+0000","last_unstale":"2024-03-26T04:41:17.724218+0000","last_undegraded":"2024-03-26T04:41:17.724218+0000","last_fullsized":"2024-03-26T04:41:17.724218+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:08:43.565470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723652+0000","last_change":"2024-03-26T04:40:48.986461+0000","last_active":"2024-03-26T04:41:17.723652+0000","last_peered":"2024-03-26T04:41:17.723652+0000","last_clean":"2024-03-26T04:41:17.723652+0000","last_became_active":"2024-03-26T04:40:48.986129+0000","last_became_peered":"2024-03-26T04:40:48.986129+0000","last_unstale":"2024-03-26T04:41:17.723652+0000","last_undegraded":"2024-03-26T04:41:17.723652+0000","last_fullsized":"2024-03-26T04:41:17.723652+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:06:58.459963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726020+0000","last_change":"2024-03-26T04:40:53.017096+0000","last_active":"2024-03-26T04:41:17.726020+0000","last_peered":"2024-03-26T04:41:17.726020+0000","last_clean":"2024-03-26T04:41:17.726020+0000","last_became_active":"2024-03-26T04:40:53.016293+0000","last_became_peered":"2024-03-26T04:40:53.016293+0000","last_unstale":"2024-03-26T04:41:17.726020+0000","last_undegraded":"2024-03-26T04:41:17.726020+0000","last_fullsized":"2024-03-26T04:41:17.726020+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:28:26.183679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725753+0000","last_change":"2024-03-26T04:40:55.026612+0000","last_active":"2024-03-26T04:41:17.725753+0000","last_peered":"2024-03-26T04:41:17.725753+0000","last_clean":"2024-03-26T04:41:17.725753+0000","last_became_active":"2024-03-26T04:40:55.026295+0000","last_became_peered":"2024-03-26T04:40:55.026295+0000","last_unstale":"2024-03-26T04:41:17.725753+0000","last_undegraded":"2024-03-26T04:41:17.725753+0000","last_fullsized":"2024-03-26T04:41:17.725753+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:42:34.832120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723408+0000","last_change":"2024-03-26T04:40:51.005813+0000","last_active":"2024-03-26T04:41:17.723408+0000","last_peered":"2024-03-26T04:41:17.723408+0000","last_clean":"2024-03-26T04:41:17.723408+0000","last_became_active":"2024-03-26T04:40:51.004468+0000","last_became_peered":"2024-03-26T04:40:51.004468+0000","last_unstale":"2024-03-26T04:41:17.723408+0000","last_undegraded":"2024-03-26T04:41:17.723408+0000","last_fullsized":"2024-03-26T04:41:17.723408+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:49:34.563695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726513+0000","last_change":"2024-03-26T04:40:48.989398+0000","last_active":"2024-03-26T04:41:17.726513+0000","last_peered":"2024-03-26T04:41:17.726513+0000","last_clean":"2024-03-26T04:41:17.726513+0000","last_became_active":"2024-03-26T04:40:48.989076+0000","last_became_peered":"2024-03-26T04:40:48.989076+0000","last_unstale":"2024-03-26T04:41:17.726513+0000","last_undegraded":"2024-03-26T04:41:17.726513+0000","last_fullsized":"2024-03-26T04:41:17.726513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:40:54.807222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:19.824638+0000","last_change":"2024-03-26T04:40:32.292330+0000","last_active":"2024-03-26T04:41:19.824638+0000","last_peered":"2024-03-26T04:41:19.824638+0000","last_clean":"2024-03-26T04:41:19.824638+0000","last_became_active":"2024-03-26T04:40:32.291982+0000","last_became_peered":"2024-03-26T04:40:32.291982+0000","last_unstale":"2024-03-26T04:41:19.824638+0000","last_undegraded":"2024-03-26T04:41:19.824638+0000","last_fullsized":"2024-03-26T04:41:19.824638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:31.279090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:31.279090+0000","last_clean_scrub_stamp":"2024-03-26T04:40:31.279090+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:46:42.399397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725662+0000","last_change":"2024-03-26T04:40:53.015246+0000","last_active":"2024-03-26T04:41:17.725662+0000","last_peered":"2024-03-26T04:41:17.725662+0000","last_clean":"2024-03-26T04:41:17.725662+0000","last_became_active":"2024-03-26T04:40:53.013906+0000","last_became_peered":"2024-03-26T04:40:53.013906+0000","last_unstale":"2024-03-26T04:41:17.725662+0000","last_undegraded":"2024-03-26T04:41:17.725662+0000","last_fullsized":"2024-03-26T04:41:17.725662+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:12:04.575786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726518+0000","last_change":"2024-03-26T04:40:55.033985+0000","last_active":"2024-03-26T04:41:17.726518+0000","last_peered":"2024-03-26T04:41:17.726518+0000","last_clean":"2024-03-26T04:41:17.726518+0000","last_became_active":"2024-03-26T04:40:55.033010+0000","last_became_peered":"2024-03-26T04:40:55.033010+0000","last_unstale":"2024-03-26T04:41:17.726518+0000","last_undegraded":"2024-03-26T04:41:17.726518+0000","last_fullsized":"2024-03-26T04:41:17.726518+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:12:00.735642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'20","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724023+0000","last_change":"2024-03-26T04:40:51.006682+0000","last_active":"2024-03-26T04:41:17.724023+0000","last_peered":"2024-03-26T04:41:17.724023+0000","last_clean":"2024-03-26T04:41:17.724023+0000","last_became_active":"2024-03-26T04:40:51.005388+0000","last_became_peered":"2024-03-26T04:40:51.005388+0000","last_unstale":"2024-03-26T04:41:17.724023+0000","last_undegraded":"2024-03-26T04:41:17.724023+0000","last_fullsized":"2024-03-26T04:41:17.724023+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:55:30.431081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724330+0000","last_change":"2024-03-26T04:40:48.987447+0000","last_active":"2024-03-26T04:41:17.724330+0000","last_peered":"2024-03-26T04:41:17.724330+0000","last_clean":"2024-03-26T04:41:17.724330+0000","last_became_active":"2024-03-26T04:40:48.986734+0000","last_became_peered":"2024-03-26T04:40:48.986734+0000","last_unstale":"2024-03-26T04:41:17.724330+0000","last_undegraded":"2024-03-26T04:41:17.724330+0000","last_fullsized":"2024-03-26T04:41:17.724330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:24:46.655209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726668+0000","last_change":"2024-03-26T04:40:53.018269+0000","last_active":"2024-03-26T04:41:17.726668+0000","last_peered":"2024-03-26T04:41:17.726668+0000","last_clean":"2024-03-26T04:41:17.726668+0000","last_became_active":"2024-03-26T04:40:53.017931+0000","last_became_peered":"2024-03-26T04:40:53.017931+0000","last_unstale":"2024-03-26T04:41:17.726668+0000","last_undegraded":"2024-03-26T04:41:17.726668+0000","last_fullsized":"2024-03-26T04:41:17.726668+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:31:32.483989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725199+0000","last_change":"2024-03-26T04:40:55.034341+0000","last_active":"2024-03-26T04:41:17.725199+0000","last_peered":"2024-03-26T04:41:17.725199+0000","last_clean":"2024-03-26T04:41:17.725199+0000","last_became_active":"2024-03-26T04:40:55.034029+0000","last_became_peered":"2024-03-26T04:40:55.034029+0000","last_unstale":"2024-03-26T04:41:17.725199+0000","last_undegraded":"2024-03-26T04:41:17.725199+0000","last_fullsized":"2024-03-26T04:41:17.725199+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:22:04.188164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727311+0000","last_change":"2024-03-26T04:40:48.985462+0000","last_active":"2024-03-26T04:41:17.727311+0000","last_peered":"2024-03-26T04:41:17.727311+0000","last_clean":"2024-03-26T04:41:17.727311+0000","last_became_active":"2024-03-26T04:40:48.985193+0000","last_became_peered":"2024-03-26T04:40:48.985193+0000","last_unstale":"2024-03-26T04:41:17.727311+0000","last_undegraded":"2024-03-26T04:41:17.727311+0000","last_fullsized":"2024-03-26T04:41:17.727311+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:14:19.269333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'34","reported_seq":93,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:29.038355+0000","last_change":"2024-03-26T04:40:51.006391+0000","last_active":"2024-03-26T04:41:29.038355+0000","last_peered":"2024-03-26T04:41:29.038355+0000","last_clean":"2024-03-26T04:41:29.038355+0000","last_became_active":"2024-03-26T04:40:51.006034+0000","last_became_peered":"2024-03-26T04:40:51.006034+0000","last_unstale":"2024-03-26T04:41:29.038355+0000","last_undegraded":"2024-03-26T04:41:29.038355+0000","last_fullsized":"2024-03-26T04:41:29.038355+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:02:50.790457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:29.038533+0000","last_change":"2024-03-26T04:40:53.018425+0000","last_active":"2024-03-26T04:41:29.038533+0000","last_peered":"2024-03-26T04:41:29.038533+0000","last_clean":"2024-03-26T04:41:29.038533+0000","last_became_active":"2024-03-26T04:40:53.018142+0000","last_became_peered":"2024-03-26T04:40:53.018142+0000","last_unstale":"2024-03-26T04:41:29.038533+0000","last_undegraded":"2024-03-26T04:41:29.038533+0000","last_fullsized":"2024-03-26T04:41:29.038533+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:02:22.793276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724340+0000","last_change":"2024-03-26T04:40:55.033821+0000","last_active":"2024-03-26T04:41:17.724340+0000","last_peered":"2024-03-26T04:41:17.724340+0000","last_clean":"2024-03-26T04:41:17.724340+0000","last_became_active":"2024-03-26T04:40:55.033515+0000","last_became_peered":"2024-03-26T04:40:55.033515+0000","last_unstale":"2024-03-26T04:41:17.724340+0000","last_undegraded":"2024-03-26T04:41:17.724340+0000","last_fullsized":"2024-03-26T04:41:17.724340+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:47:08.857191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725272+0000","last_change":"2024-03-26T04:40:48.985955+0000","last_active":"2024-03-26T04:41:17.725272+0000","last_peered":"2024-03-26T04:41:17.725272+0000","last_clean":"2024-03-26T04:41:17.725272+0000","last_became_active":"2024-03-26T04:40:48.985466+0000","last_became_peered":"2024-03-26T04:40:48.985466+0000","last_unstale":"2024-03-26T04:41:17.725272+0000","last_undegraded":"2024-03-26T04:41:17.725272+0000","last_fullsized":"2024-03-26T04:41:17.725272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:53:26.081275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726246+0000","last_change":"2024-03-26T04:40:51.007875+0000","last_active":"2024-03-26T04:41:17.726246+0000","last_peered":"2024-03-26T04:41:17.726246+0000","last_clean":"2024-03-26T04:41:17.726246+0000","last_became_active":"2024-03-26T04:40:51.006576+0000","last_became_peered":"2024-03-26T04:40:51.006576+0000","last_unstale":"2024-03-26T04:41:17.726246+0000","last_undegraded":"2024-03-26T04:41:17.726246+0000","last_fullsized":"2024-03-26T04:41:17.726246+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:35:04.283711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:29.038354+0000","last_change":"2024-03-26T04:40:53.016891+0000","last_active":"2024-03-26T04:41:29.038354+0000","last_peered":"2024-03-26T04:41:29.038354+0000","last_clean":"2024-03-26T04:41:29.038354+0000","last_became_active":"2024-03-26T04:40:53.016603+0000","last_became_peered":"2024-03-26T04:40:53.016603+0000","last_unstale":"2024-03-26T04:41:29.038354+0000","last_undegraded":"2024-03-26T04:41:29.038354+0000","last_fullsized":"2024-03-26T04:41:29.038354+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:06:22.897370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724824+0000","last_change":"2024-03-26T04:40:55.033463+0000","last_active":"2024-03-26T04:41:17.724824+0000","last_peered":"2024-03-26T04:41:17.724824+0000","last_clean":"2024-03-26T04:41:17.724824+0000","last_became_active":"2024-03-26T04:40:55.033210+0000","last_became_peered":"2024-03-26T04:40:55.033210+0000","last_unstale":"2024-03-26T04:41:17.724824+0000","last_undegraded":"2024-03-26T04:41:17.724824+0000","last_fullsized":"2024-03-26T04:41:17.724824+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:16:17.631012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726881+0000","last_change":"2024-03-26T04:40:48.988283+0000","last_active":"2024-03-26T04:41:17.726881+0000","last_peered":"2024-03-26T04:41:17.726881+0000","last_clean":"2024-03-26T04:41:17.726881+0000","last_became_active":"2024-03-26T04:40:48.985904+0000","last_became_peered":"2024-03-26T04:40:48.985904+0000","last_unstale":"2024-03-26T04:41:17.726881+0000","last_undegraded":"2024-03-26T04:41:17.726881+0000","last_fullsized":"2024-03-26T04:41:17.726881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:24:40.555792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725081+0000","last_change":"2024-03-26T04:40:51.006111+0000","last_active":"2024-03-26T04:41:17.725081+0000","last_peered":"2024-03-26T04:41:17.725081+0000","last_clean":"2024-03-26T04:41:17.725081+0000","last_became_active":"2024-03-26T04:40:51.004395+0000","last_became_peered":"2024-03-26T04:40:51.004395+0000","last_unstale":"2024-03-26T04:41:17.725081+0000","last_undegraded":"2024-03-26T04:41:17.725081+0000","last_fullsized":"2024-03-26T04:41:17.725081+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:18:33.429560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724466+0000","last_change":"2024-03-26T04:40:53.013418+0000","last_active":"2024-03-26T04:41:17.724466+0000","last_peered":"2024-03-26T04:41:17.724466+0000","last_clean":"2024-03-26T04:41:17.724466+0000","last_became_active":"2024-03-26T04:40:53.013142+0000","last_became_peered":"2024-03-26T04:40:53.013142+0000","last_unstale":"2024-03-26T04:41:17.724466+0000","last_undegraded":"2024-03-26T04:41:17.724466+0000","last_fullsized":"2024-03-26T04:41:17.724466+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:11:12.747099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726663+0000","last_change":"2024-03-26T04:40:55.033455+0000","last_active":"2024-03-26T04:41:17.726663+0000","last_peered":"2024-03-26T04:41:17.726663+0000","last_clean":"2024-03-26T04:41:17.726663+0000","last_became_active":"2024-03-26T04:40:55.033013+0000","last_became_peered":"2024-03-26T04:40:55.033013+0000","last_unstale":"2024-03-26T04:41:17.726663+0000","last_undegraded":"2024-03-26T04:41:17.726663+0000","last_fullsized":"2024-03-26T04:41:17.726663+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:35:56.434531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:29.038490+0000","last_change":"2024-03-26T04:40:53.014667+0000","last_active":"2024-03-26T04:41:29.038490+0000","last_peered":"2024-03-26T04:41:29.038490+0000","last_clean":"2024-03-26T04:41:29.038490+0000","last_became_active":"2024-03-26T04:40:53.014278+0000","last_became_peered":"2024-03-26T04:40:53.014278+0000","last_unstale":"2024-03-26T04:41:29.038490+0000","last_undegraded":"2024-03-26T04:41:29.038490+0000","last_fullsized":"2024-03-26T04:41:29.038490+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:02:33.124733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725263+0000","last_change":"2024-03-26T04:40:48.987055+0000","last_active":"2024-03-26T04:41:17.725263+0000","last_peered":"2024-03-26T04:41:17.725263+0000","last_clean":"2024-03-26T04:41:17.725263+0000","last_became_active":"2024-03-26T04:40:48.985073+0000","last_became_peered":"2024-03-26T04:40:48.985073+0000","last_unstale":"2024-03-26T04:41:17.725263+0000","last_undegraded":"2024-03-26T04:41:17.725263+0000","last_fullsized":"2024-03-26T04:41:17.725263+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:59:19.348919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'24","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725013+0000","last_change":"2024-03-26T04:40:51.009152+0000","last_active":"2024-03-26T04:41:17.725013+0000","last_peered":"2024-03-26T04:41:17.725013+0000","last_clean":"2024-03-26T04:41:17.725013+0000","last_became_active":"2024-03-26T04:40:51.008792+0000","last_became_peered":"2024-03-26T04:40:51.008792+0000","last_unstale":"2024-03-26T04:41:17.725013+0000","last_undegraded":"2024-03-26T04:41:17.725013+0000","last_fullsized":"2024-03-26T04:41:17.725013+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:31:14.276280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726859+0000","last_change":"2024-03-26T04:40:55.027709+0000","last_active":"2024-03-26T04:41:17.726859+0000","last_peered":"2024-03-26T04:41:17.726859+0000","last_clean":"2024-03-26T04:41:17.726859+0000","last_became_active":"2024-03-26T04:40:55.027382+0000","last_became_peered":"2024-03-26T04:40:55.027382+0000","last_unstale":"2024-03-26T04:41:17.726859+0000","last_undegraded":"2024-03-26T04:41:17.726859+0000","last_fullsized":"2024-03-26T04:41:17.726859+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:33:40.657766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725928+0000","last_change":"2024-03-26T04:40:53.015658+0000","last_active":"2024-03-26T04:41:17.725928+0000","last_peered":"2024-03-26T04:41:17.725928+0000","last_clean":"2024-03-26T04:41:17.725928+0000","last_became_active":"2024-03-26T04:40:53.015194+0000","last_became_peered":"2024-03-26T04:40:53.015194+0000","last_unstale":"2024-03-26T04:41:17.725928+0000","last_undegraded":"2024-03-26T04:41:17.725928+0000","last_fullsized":"2024-03-26T04:41:17.725928+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:08:55.582534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723487+0000","last_change":"2024-03-26T04:40:48.987423+0000","last_active":"2024-03-26T04:41:17.723487+0000","last_peered":"2024-03-26T04:41:17.723487+0000","last_clean":"2024-03-26T04:41:17.723487+0000","last_became_active":"2024-03-26T04:40:48.986511+0000","last_became_peered":"2024-03-26T04:40:48.986511+0000","last_unstale":"2024-03-26T04:41:17.723487+0000","last_undegraded":"2024-03-26T04:41:17.723487+0000","last_fullsized":"2024-03-26T04:41:17.723487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:05:49.035782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'27","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724611+0000","last_change":"2024-03-26T04:40:51.007321+0000","last_active":"2024-03-26T04:41:17.724611+0000","last_peered":"2024-03-26T04:41:17.724611+0000","last_clean":"2024-03-26T04:41:17.724611+0000","last_became_active":"2024-03-26T04:40:51.007012+0000","last_became_peered":"2024-03-26T04:40:51.007012+0000","last_unstale":"2024-03-26T04:41:17.724611+0000","last_undegraded":"2024-03-26T04:41:17.724611+0000","last_fullsized":"2024-03-26T04:41:17.724611+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:22:37.763876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726660+0000","last_change":"2024-03-26T04:40:55.026964+0000","last_active":"2024-03-26T04:41:17.726660+0000","last_peered":"2024-03-26T04:41:17.726660+0000","last_clean":"2024-03-26T04:41:17.726660+0000","last_became_active":"2024-03-26T04:40:55.026499+0000","last_became_peered":"2024-03-26T04:40:55.026499+0000","last_unstale":"2024-03-26T04:41:17.726660+0000","last_undegraded":"2024-03-26T04:41:17.726660+0000","last_fullsized":"2024-03-26T04:41:17.726660+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:13:57.444687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725459+0000","last_change":"2024-03-26T04:40:48.984998+0000","last_active":"2024-03-26T04:41:17.725459+0000","last_peered":"2024-03-26T04:41:17.725459+0000","last_clean":"2024-03-26T04:41:17.725459+0000","last_became_active":"2024-03-26T04:40:48.984638+0000","last_became_peered":"2024-03-26T04:40:48.984638+0000","last_unstale":"2024-03-26T04:41:17.725459+0000","last_undegraded":"2024-03-26T04:41:17.725459+0000","last_fullsized":"2024-03-26T04:41:17.725459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:23:31.624155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723408+0000","last_change":"2024-03-26T04:40:51.005732+0000","last_active":"2024-03-26T04:41:17.723408+0000","last_peered":"2024-03-26T04:41:17.723408+0000","last_clean":"2024-03-26T04:41:17.723408+0000","last_became_active":"2024-03-26T04:40:51.004548+0000","last_became_peered":"2024-03-26T04:40:51.004548+0000","last_unstale":"2024-03-26T04:41:17.723408+0000","last_undegraded":"2024-03-26T04:41:17.723408+0000","last_fullsized":"2024-03-26T04:41:17.723408+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:25:11.762176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725416+0000","last_change":"2024-03-26T04:40:53.013931+0000","last_active":"2024-03-26T04:41:17.725416+0000","last_peered":"2024-03-26T04:41:17.725416+0000","last_clean":"2024-03-26T04:41:17.725416+0000","last_became_active":"2024-03-26T04:40:53.013508+0000","last_became_peered":"2024-03-26T04:40:53.013508+0000","last_unstale":"2024-03-26T04:41:17.725416+0000","last_undegraded":"2024-03-26T04:41:17.725416+0000","last_fullsized":"2024-03-26T04:41:17.725416+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:48:36.939752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724266+0000","last_change":"2024-03-26T04:40:55.036307+0000","last_active":"2024-03-26T04:41:17.724266+0000","last_peered":"2024-03-26T04:41:17.724266+0000","last_clean":"2024-03-26T04:41:17.724266+0000","last_became_active":"2024-03-26T04:40:55.035188+0000","last_became_peered":"2024-03-26T04:40:55.035188+0000","last_unstale":"2024-03-26T04:41:17.724266+0000","last_undegraded":"2024-03-26T04:41:17.724266+0000","last_fullsized":"2024-03-26T04:41:17.724266+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:48:28.647161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:29.038480+0000","last_change":"2024-03-26T04:40:53.015225+0000","last_active":"2024-03-26T04:41:29.038480+0000","last_peered":"2024-03-26T04:41:29.038480+0000","last_clean":"2024-03-26T04:41:29.038480+0000","last_became_active":"2024-03-26T04:40:53.014628+0000","last_became_peered":"2024-03-26T04:40:53.014628+0000","last_unstale":"2024-03-26T04:41:29.038480+0000","last_undegraded":"2024-03-26T04:41:29.038480+0000","last_fullsized":"2024-03-26T04:41:29.038480+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:53:42.384096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.725316+0000","last_change":"2024-03-26T04:40:48.988583+0000","last_active":"2024-03-26T04:41:17.725316+0000","last_peered":"2024-03-26T04:41:17.725316+0000","last_clean":"2024-03-26T04:41:17.725316+0000","last_became_active":"2024-03-26T04:40:48.988186+0000","last_became_peered":"2024-03-26T04:40:48.988186+0000","last_unstale":"2024-03-26T04:41:17.725316+0000","last_undegraded":"2024-03-26T04:41:17.725316+0000","last_fullsized":"2024-03-26T04:41:17.725316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:27:24.753635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'25","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726361+0000","last_change":"2024-03-26T04:40:51.005929+0000","last_active":"2024-03-26T04:41:17.726361+0000","last_peered":"2024-03-26T04:41:17.726361+0000","last_clean":"2024-03-26T04:41:17.726361+0000","last_became_active":"2024-03-26T04:40:51.005424+0000","last_became_peered":"2024-03-26T04:40:51.005424+0000","last_unstale":"2024-03-26T04:41:17.726361+0000","last_undegraded":"2024-03-26T04:41:17.726361+0000","last_fullsized":"2024-03-26T04:41:17.726361+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:24:03.491340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724780+0000","last_change":"2024-03-26T04:40:55.027288+0000","last_active":"2024-03-26T04:41:17.724780+0000","last_peered":"2024-03-26T04:41:17.724780+0000","last_clean":"2024-03-26T04:41:17.724780+0000","last_became_active":"2024-03-26T04:40:55.027010+0000","last_became_peered":"2024-03-26T04:40:55.027010+0000","last_unstale":"2024-03-26T04:41:17.724780+0000","last_undegraded":"2024-03-26T04:41:17.724780+0000","last_fullsized":"2024-03-26T04:41:17.724780+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:09:45.859513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726335+0000","last_change":"2024-03-26T04:40:48.989060+0000","last_active":"2024-03-26T04:41:17.726335+0000","last_peered":"2024-03-26T04:41:17.726335+0000","last_clean":"2024-03-26T04:41:17.726335+0000","last_became_active":"2024-03-26T04:40:48.988487+0000","last_became_peered":"2024-03-26T04:40:48.988487+0000","last_unstale":"2024-03-26T04:41:17.726335+0000","last_undegraded":"2024-03-26T04:41:17.726335+0000","last_fullsized":"2024-03-26T04:41:17.726335+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:38:19.861083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727562+0000","last_change":"2024-03-26T04:40:51.003025+0000","last_active":"2024-03-26T04:41:17.727562+0000","last_peered":"2024-03-26T04:41:17.727562+0000","last_clean":"2024-03-26T04:41:17.727562+0000","last_became_active":"2024-03-26T04:40:51.002617+0000","last_became_peered":"2024-03-26T04:40:51.002617+0000","last_unstale":"2024-03-26T04:41:17.727562+0000","last_undegraded":"2024-03-26T04:41:17.727562+0000","last_fullsized":"2024-03-26T04:41:17.727562+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:51:57.458352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724957+0000","last_change":"2024-03-26T04:40:53.014635+0000","last_active":"2024-03-26T04:41:17.724957+0000","last_peered":"2024-03-26T04:41:17.724957+0000","last_clean":"2024-03-26T04:41:17.724957+0000","last_became_active":"2024-03-26T04:40:53.014278+0000","last_became_peered":"2024-03-26T04:40:53.014278+0000","last_unstale":"2024-03-26T04:41:17.724957+0000","last_undegraded":"2024-03-26T04:41:17.724957+0000","last_fullsized":"2024-03-26T04:41:17.724957+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:04:08.408265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.722656+0000","last_change":"2024-03-26T04:40:55.036515+0000","last_active":"2024-03-26T04:41:17.722656+0000","last_peered":"2024-03-26T04:41:17.722656+0000","last_clean":"2024-03-26T04:41:17.722656+0000","last_became_active":"2024-03-26T04:40:55.036191+0000","last_became_peered":"2024-03-26T04:40:55.036191+0000","last_unstale":"2024-03-26T04:41:17.722656+0000","last_undegraded":"2024-03-26T04:41:17.722656+0000","last_fullsized":"2024-03-26T04:41:17.722656+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:25:45.972122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.727011+0000","last_change":"2024-03-26T04:40:48.988798+0000","last_active":"2024-03-26T04:41:17.727011+0000","last_peered":"2024-03-26T04:41:17.727011+0000","last_clean":"2024-03-26T04:41:17.727011+0000","last_became_active":"2024-03-26T04:40:48.988050+0000","last_became_peered":"2024-03-26T04:40:48.988050+0000","last_unstale":"2024-03-26T04:41:17.727011+0000","last_undegraded":"2024-03-26T04:41:17.727011+0000","last_fullsized":"2024-03-26T04:41:17.727011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:25:39.400286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726108+0000","last_change":"2024-03-26T04:40:51.003895+0000","last_active":"2024-03-26T04:41:17.726108+0000","last_peered":"2024-03-26T04:41:17.726108+0000","last_clean":"2024-03-26T04:41:17.726108+0000","last_became_active":"2024-03-26T04:40:51.003363+0000","last_became_peered":"2024-03-26T04:40:51.003363+0000","last_unstale":"2024-03-26T04:41:17.726108+0000","last_undegraded":"2024-03-26T04:41:17.726108+0000","last_fullsized":"2024-03-26T04:41:17.726108+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:02:03.463355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.723429+0000","last_change":"2024-03-26T04:40:53.016083+0000","last_active":"2024-03-26T04:41:17.723429+0000","last_peered":"2024-03-26T04:41:17.723429+0000","last_clean":"2024-03-26T04:41:17.723429+0000","last_became_active":"2024-03-26T04:40:53.015594+0000","last_became_peered":"2024-03-26T04:40:53.015594+0000","last_unstale":"2024-03-26T04:41:17.723429+0000","last_undegraded":"2024-03-26T04:41:17.723429+0000","last_fullsized":"2024-03-26T04:41:17.723429+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:17:17.273202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724522+0000","last_change":"2024-03-26T04:40:55.033175+0000","last_active":"2024-03-26T04:41:17.724522+0000","last_peered":"2024-03-26T04:41:17.724522+0000","last_clean":"2024-03-26T04:41:17.724522+0000","last_became_active":"2024-03-26T04:40:55.032780+0000","last_became_peered":"2024-03-26T04:40:55.032780+0000","last_unstale":"2024-03-26T04:41:17.724522+0000","last_undegraded":"2024-03-26T04:41:17.724522+0000","last_fullsized":"2024-03-26T04:41:17.724522+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:54.014039+0000","last_clean_scrub_stamp":"2024-03-26T04:40:54.014039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:41:40.707665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724891+0000","last_change":"2024-03-26T04:40:48.986308+0000","last_active":"2024-03-26T04:41:17.724891+0000","last_peered":"2024-03-26T04:41:17.724891+0000","last_clean":"2024-03-26T04:41:17.724891+0000","last_became_active":"2024-03-26T04:40:48.985995+0000","last_became_peered":"2024-03-26T04:40:48.985995+0000","last_unstale":"2024-03-26T04:41:17.724891+0000","last_undegraded":"2024-03-26T04:41:17.724891+0000","last_fullsized":"2024-03-26T04:41:17.724891+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:47.956970+0000","last_clean_scrub_stamp":"2024-03-26T04:40:47.956970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:26:07.709967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.726923+0000","last_change":"2024-03-26T04:40:51.006941+0000","last_active":"2024-03-26T04:41:17.726923+0000","last_peered":"2024-03-26T04:41:17.726923+0000","last_clean":"2024-03-26T04:41:17.726923+0000","last_became_active":"2024-03-26T04:40:51.006559+0000","last_became_peered":"2024-03-26T04:40:51.006559+0000","last_unstale":"2024-03-26T04:41:17.726923+0000","last_undegraded":"2024-03-26T04:41:17.726923+0000","last_fullsized":"2024-03-26T04:41:17.726923+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:49.975670+0000","last_clean_scrub_stamp":"2024-03-26T04:40:49.975670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:14:56.406226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:41:17.724715+0000","last_change":"2024-03-26T04:40:53.013691+0000","last_active":"2024-03-26T04:41:17.724715+0000","last_peered":"2024-03-26T04:41:17.724715+0000","last_clean":"2024-03-26T04:41:17.724715+0000","last_became_active":"2024-03-26T04:40:53.013417+0000","last_became_peered":"2024-03-26T04:40:53.013417+0000","last_unstale":"2024-03-26T04:41:17.724715+0000","last_undegraded":"2024-03-26T04:41:17.724715+0000","last_fullsized":"2024-03-26T04:41:17.724715+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:40:51.994529+0000","last_clean_scrub_stamp":"2024-03-26T04:40:51.994529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:38:25.683203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":587,"ondisk_log_size":587,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411344,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738394,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:41:42.781 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T04:41:42.781 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T04:41:42.781 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T04:41:42.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T04:41:43.237 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:43.299 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:41:43.299 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:43.299 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.299 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:41:43.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:41:43.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:41:43.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: Deploying daemon grafana.a on smithi114 2024-03-26T04:41:43.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:43 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:41:43.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:43.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:41:43.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:41:43.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:41:43.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: Deploying daemon grafana.a on smithi114 2024-03-26T04:41:43.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:43 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:41:43.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:43.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:41:43.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:41:43.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:41:43.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:43.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: Deploying daemon grafana.a on smithi114 2024-03-26T04:41:43.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:43 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:44.288 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:44 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[49188]: ts=2024-03-26T04:41:44.068Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001087677s 2024-03-26T04:41:45.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:45 smithi038 ceph-mon[25836]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:45.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:45 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:45.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:45 smithi114 ceph-mon[27627]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:45.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:45 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:45.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:45 smithi177 ceph-mon[27677]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:45.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:45 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:47.120 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T04:41:47.120 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T04:41:47.532 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:47 smithi177 ceph-mon[27677]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:47.532 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:47 smithi177 ceph-mon[27677]: from='client.? 172.21.15.38:0/3527736650' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:41:47.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:47 smithi038 ceph-mon[25836]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:47.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:47 smithi038 ceph-mon[25836]: from='client.? 172.21.15.38:0/3527736650' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:41:47.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:47 smithi114 ceph-mon[27627]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:47.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:47 smithi114 ceph-mon[27627]: from='client.? 172.21.15.38:0/3527736650' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:41:48.013 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T04:41:48.014 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T04:41:48.014 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T04:41:48.026 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T04:41:48.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-26T04:41:48.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:48 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:48.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:48 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:48.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:48 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:48.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:48 smithi114 ceph-mon[27627]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:48.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:48 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:48.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:48 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:48.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:48 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:48.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:48 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:48.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:48 smithi177 ceph-mon[27677]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:48.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:48 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:49.009 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:48 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:49.009 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:48 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:49.009 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:48 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:49.009 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:48 smithi038 ceph-mon[25836]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:49.009 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:48 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:49.330 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:49 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: ::ffff:172.21.15.114 - - [26/Mar/2024:04:41:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:41:49.343 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:50.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:50 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:50.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:50 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:50.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:50 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:50.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:50 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:50.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:50 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:50.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:50 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:50.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:50 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:50.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:50 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:50.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:50 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:50.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:50 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:50.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:50 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:50.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:50 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:50.988 INFO:teuthology.orchestra.run.smithi038.stdout:167 167 2024-03-26T04:41:51.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:51 smithi038 ceph-mon[25836]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:51.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:51 smithi177 ceph-mon[27677]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:51.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:51 smithi114 ceph-mon[27627]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:51.756 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-26T04:41:52.331 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:41:52 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[49188]: ts=2024-03-26T04:41:52.069Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002498878s 2024-03-26T04:41:52.594 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:53.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:53 smithi038 ceph-mon[25836]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:53.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:53 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:53.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:53 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:53.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:53 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:53.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:53 smithi114 ceph-mon[27627]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:53.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:53 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:53.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:53 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:53.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:53 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:53.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:53 smithi177 ceph-mon[27677]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:53.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:53 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:53.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:53 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:53.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:53 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:54.582 INFO:teuthology.orchestra.run.smithi038.stdout:Backend: cephadm 2024-03-26T04:41:54.583 INFO:teuthology.orchestra.run.smithi038.stdout:Available: Yes 2024-03-26T04:41:54.583 INFO:teuthology.orchestra.run.smithi038.stdout:Paused: No 2024-03-26T04:41:55.330 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:55 smithi038 ceph-mon[25836]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:55.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-mon[27627]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:55.396 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 systemd[1]: Starting Ceph grafana.a for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:41:55.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-26T04:41:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:55.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:55 smithi177 ceph-mon[27677]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:55.767 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 podman[38353]: 2024-03-26 04:41:55.349662063 +0000 UTC m=+0.017003828 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-26T04:41:55.767 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 podman[38353]: 2024-03-26 04:41:55.449105626 +0000 UTC m=+0.116447437 container create acfb516583bf849100cc55822035d7bf16b2441106fcbffdfd9e50fafdbbe6d4 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a, architecture=x86_64, io.openshift.expose-services=, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, distribution-scope=public, io.openshift.tags=base rhel8, description=Ceph Grafana Container, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , name=ubi8) 2024-03-26T04:41:55.767 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 podman[38353]: 2024-03-26 04:41:55.676217808 +0000 UTC m=+0.343559576 container init acfb516583bf849100cc55822035d7bf16b2441106fcbffdfd9e50fafdbbe6d4 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.buildah.version=1.32.0, com.redhat.component=ubi8-container, version=8.5, release=239.1651231664, architecture=x86_64, vcs-type=git, description=Ceph Grafana Container, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-26T04:41:55.767 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 podman[38353]: 2024-03-26 04:41:55.682073684 +0000 UTC m=+0.349415450 container start acfb516583bf849100cc55822035d7bf16b2441106fcbffdfd9e50fafdbbe6d4 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a, io.buildah.version=1.32.0, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, name=ubi8, distribution-scope=public, version=8.5) 2024-03-26T04:41:56.017 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 bash[38353]: acfb516583bf849100cc55822035d7bf16b2441106fcbffdfd9e50fafdbbe6d4 2024-03-26T04:41:56.017 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 systemd[1]: Started Ceph grafana.a for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:41:56.017 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.834109731Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T04:41:56.017 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.83434327Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T04:41:56.018 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.834373772Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T04:41:56.018 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.834385896Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T04:41:56.018 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.834397036Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T04:41:56.018 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.834407049Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T04:41:56.018 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.834418318Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T04:41:56.018 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.834428284Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T04:41:56.018 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.834438828Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T04:41:56.018 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.834476423Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T04:41:56.018 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.834487269Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T04:41:56.018 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.834497293Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T04:41:56.018 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.834506934Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T04:41:56.018 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=settings t=2024-03-26T04:41:55.834516589Z level=info msg="App mode production" 2024-03-26T04:41:56.019 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=sqlstore t=2024-03-26T04:41:55.834595239Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T04:41:56.019 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=sqlstore t=2024-03-26T04:41:55.83462073Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T04:41:56.019 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.835762759Z level=info msg="Starting DB migrations" 2024-03-26T04:41:56.019 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.835855214Z level=info msg="Executing migration" id="create migration_log table" 2024-03-26T04:41:56.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.836400361Z level=info msg="Executing migration" id="create user table" 2024-03-26T04:41:56.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.836926097Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-26T04:41:56.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.837386805Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-26T04:41:56.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.837838995Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-26T04:41:56.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.838280887Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-26T04:41:56.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.838637278Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-26T04:41:56.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.83936749Z level=info msg="Executing migration" id="create user table v2" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.839767157Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.840197434Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.840624517Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.840981884Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.841306787Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.841684509Z level=info msg="Executing migration" id="Update user table charset" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.841988078Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.842399669Z level=info msg="Executing migration" id="Add missing user data" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.842740912Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.843173024Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.84359526Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.844042419Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.845221811Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-26T04:41:56.021 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.845689347Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-26T04:41:56.022 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.846145488Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T04:41:56.022 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.847220595Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-26T04:41:56.022 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.847623469Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-26T04:41:56.022 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.848045936Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-26T04:41:56.022 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.848352433Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-26T04:41:56.022 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.848721171Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-26T04:41:56.022 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.849125779Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-26T04:41:56.022 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.849491382Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-26T04:41:56.022 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.849849901Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T04:41:56.022 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.850484778Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-26T04:41:56.022 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.850856984Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-26T04:41:56.022 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.851259574Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-26T04:41:56.022 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.851655064Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.852057284Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.852473915Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.852814162Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.853197562Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.853526986Z level=info msg="Executing migration" id="create star table" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.853878478Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.854288749Z level=info msg="Executing migration" id="create org table v1" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.854675274Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.857445808Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.860326712Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.862997925Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.863411236Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.86631398Z level=info msg="Executing migration" id="Update org table charset" 2024-03-26T04:41:56.023 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.868901671Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.871707631Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.872069468Z level=info msg="Executing migration" id="create dashboard table" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.874833815Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.877532348Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.880294307Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.880864614Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.883613436Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.886220374Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.888993285Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.889420115Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.892004068Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.894865148Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.895798239Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.896295827Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-26T04:41:56.024 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.897131421Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.897703499Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.898361854Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.899087549Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.899506425Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.900348378Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.901272271Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.90198948Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.902307396Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.903131623Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.903689238Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.90426251Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.905092662Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.905594278Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.906597077Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-26T04:41:56.025 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.907292147Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.908199624Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.908507127Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.909437973Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.910377506Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.91139413Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.911837833Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.912704135Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.913526509Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.914396944Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.915225189Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.916051491Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.916991406Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.917719622Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-26T04:41:56.026 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.918059715Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.918907243Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.919971015Z level=info msg="Executing migration" id="create data_source table" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.920750852Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.921362783Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.924198871Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.927039958Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.929956083Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.931103482Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.933964385Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.936698869Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.939636457Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.940037343Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.942805977Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-26T04:41:56.027 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.9456503Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.948521646Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.948894452Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.951845289Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.954616716Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.957402285Z level=info msg="Executing migration" id="Add uid column" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.957948768Z level=info msg="Executing migration" id="Update uid value" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.960742746Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.963521793Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.966315448Z level=info msg="Executing migration" id="create api_key table" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.966877325Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.969713737Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.972516885Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.975174205Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-26T04:41:56.028 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.977714708Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.98050993Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.980984882Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.983752504Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.986399604Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.98911187Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.98954699Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.992419418Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.995023693Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.997608524Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.997942758Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.998919879Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:55 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:55.99964353Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.000627025Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.001317196Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T04:41:56.029 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.002275208Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-26T04:41:56.030 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.002827188Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-26T04:41:56.030 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.003206739Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-26T04:41:56.030 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.004016633Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T04:41:56.030 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.004481301Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T04:41:56.030 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.004895401Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T04:41:56.030 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.005740685Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T04:41:56.030 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.006573922Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.007495735Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.008327713Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.008877746Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.009726489Z level=info msg="Executing migration" id="create quota table v1" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.010594667Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.011415204Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.011712094Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.012727559Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.013495749Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.014358399Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.014675397Z level=info msg="Executing migration" id="create session table" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.015655404Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.016390645Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-26T04:41:56.031 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.017310953Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-26T04:41:56.032 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.01789543Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-26T04:41:56.032 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.018310624Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-26T04:41:56.032 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.019159662Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-26T04:41:56.032 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.019473882Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-26T04:41:56.032 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.020006499Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-26T04:41:56.032 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.020313881Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-26T04:41:56.032 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.020888573Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-26T04:41:56.032 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.021208385Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-26T04:41:56.268 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-mon[27627]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:56.268 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:56.268 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:56.268 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:56.268 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:56.269 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.022042966Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-26T04:41:56.269 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.022393718Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-26T04:41:56.269 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.024873399Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T04:41:56.269 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.027783588Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T04:41:56.269 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.028117355Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T04:41:56.269 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.030936827Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T04:41:56.269 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.033879Z level=info msg="Executing migration" id="create alert table v1" 2024-03-26T04:41:56.269 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.036724729Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-26T04:41:56.269 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.037218939Z level=info msg="Executing migration" id="add index alert state" 2024-03-26T04:41:56.269 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.039955841Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-26T04:41:56.269 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.042979149Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-26T04:41:56.269 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.045669598Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T04:41:56.269 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.046256165Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T04:41:56.270 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.046647854Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T04:41:56.270 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.048057062Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-26T04:41:56.270 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.04845346Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T04:41:56.270 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.048881796Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-26T04:41:56.270 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.049223629Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-26T04:41:56.270 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.049570261Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-26T04:41:56.270 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.04997987Z level=info msg="Executing migration" id="Add column is_default" 2024-03-26T04:41:56.270 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.050606835Z level=info msg="Executing migration" id="Add column frequency" 2024-03-26T04:41:56.270 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.051301601Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-26T04:41:56.270 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.051994232Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-26T04:41:56.270 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.052638519Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-26T04:41:56.270 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.053083165Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-26T04:41:56.270 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.05339114Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.053677767Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.05407855Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.054510007Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.054915606Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.055829589Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.056280579Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.057160194Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.057988808Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.058986614Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.059634996Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.060505658Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.061328966Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.064143373Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T04:41:56.271 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.064592662Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.067278847Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.070081205Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.073053635Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.073500436Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.076435343Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.079337718Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.082148058Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.082652597Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.085596224Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.088398Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.091151559Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.091878989Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.094728989Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.097683485Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T04:41:56.272 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.100323938Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T04:41:56.273 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.101908079Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-26T04:41:56.273 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.104720592Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T04:41:56.274 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.107318029Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-26T04:41:56.274 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.110332909Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-26T04:41:56.274 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.110855272Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-26T04:41:56.274 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.113640268Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-26T04:41:56.274 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.116286157Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-26T04:41:56.274 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.118977632Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-26T04:41:56.274 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.11941472Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-26T04:41:56.274 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.122169887Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-26T04:41:56.274 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.124804255Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-26T04:41:56.274 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.127535217Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.12800341Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.130840442Z level=info msg="Executing migration" id="Move region to single row" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.133535457Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.136321584Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.136779432Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.139601442Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.140257285Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.141090419Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.141527243Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.14235898Z level=info msg="Executing migration" id="create test_data table" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.142792344Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.143213199Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.14403074Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.144471044Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-26T04:41:56.275 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.145274999Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.145618359Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.146243267Z level=info msg="Executing migration" id="create team table" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.146974828Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.14742873Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.148446705Z level=info msg="Executing migration" id="create team member table" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.149090667Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.150055346Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.150573609Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.151424643Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.152280334Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.153115749Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.153838085Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.154843832Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.155548858Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T04:41:56.276 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.156025954Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.156493757Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.157325835Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.158144855Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.158590392Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.159014756Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.15941386Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.159710301Z level=info msg="Executing migration" id="create tag table" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.160116683Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.160556587Z level=info msg="Executing migration" id="create login attempt table" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.160950127Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.161366419Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.161760505Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.163630136Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.164069923Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.164510308Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-26T04:41:56.277 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.164814317Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-26T04:41:56.278 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.165193576Z level=info msg="Executing migration" id="create user auth table" 2024-03-26T04:41:56.278 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.165572572Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T04:41:56.278 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.166026691Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-26T04:41:56.278 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.166327291Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-26T04:41:56.278 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.167066703Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-26T04:41:56.278 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.167832466Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-26T04:41:56.278 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.169051334Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-26T04:41:56.278 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.169823915Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-26T04:41:56.278 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.170258837Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T04:41:56.278 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.170999553Z level=info msg="Executing migration" id="create server_lock table" 2024-03-26T04:41:56.278 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.171393158Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-26T04:41:56.278 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.171830494Z level=info msg="Executing migration" id="create user auth token table" 2024-03-26T04:41:56.278 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.172299494Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.17271836Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.173162102Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.173584273Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.174350165Z level=info msg="Executing migration" id="create cache_data table" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.174764639Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.175233715Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.17565321Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.176095567Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.176398247Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.17668401Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.177126008Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.177549322Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.177982856Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.178970413Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-26T04:41:56.279 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.179707387Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.18271076Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.183227833Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.186177926Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.189003562Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.191710638Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.192045068Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.194901798Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.19536688Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.1978331Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.198148775Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.200700675Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.2013841Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.204253884Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.20469557Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.207146148Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T04:41:56.280 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.210041369Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.212830492Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.216262937Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.21960474Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.222272555Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.225060597Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.225988156Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.228616575Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.231374379Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.233899838Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.234335117Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.237049171Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.237996727Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.239804411Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.242635229Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.243089325Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.245900678Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-26T04:41:56.281 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.248578152Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-26T04:41:56.282 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.251284212Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T04:41:56.282 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.251744136Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T04:41:56.282 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.252635116Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T04:41:56.282 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.253485439Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T04:41:56.282 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.254333214Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-26T04:41:56.282 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.254816066Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T04:41:56.282 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.255668976Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T04:41:56.282 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.25651911Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T04:41:56.282 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.257441807Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-26T04:41:56.282 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.258415524Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-26T04:41:56.282 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.259311561Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-26T04:41:56.282 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.260225297Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T04:41:56.282 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.261074553Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.261911503Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.262935898Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.263638812Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.264514009Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.264829757Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.26574451Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.266541438Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.26745599Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.267865349Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.268752331Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.269654143Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.270494512Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.270968902Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.271859016Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T04:41:56.283 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.272684659Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T04:41:56.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:56 smithi038 ceph-mon[25836]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:56.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:56 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:56.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:56 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:56.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:56 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:56.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:56 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:56.542 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.27353492Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T04:41:56.542 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.273984295Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T04:41:56.542 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.27482915Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T04:41:56.542 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.274929927Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T04:41:56.542 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.27569669Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T04:41:56.542 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.276568283Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T04:41:56.542 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.27704807Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-26T04:41:56.542 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.277190618Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-26T04:41:56.542 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.277324199Z level=info msg="alerts found to migrate" alerts=0 2024-03-26T04:41:56.542 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.277898483Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-26T04:41:56.543 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.278741664Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-26T04:41:56.543 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.279590943Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-26T04:41:56.543 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.28002587Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T04:41:56.543 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.282865762Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-26T04:41:56.543 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.285740132Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T04:41:56.543 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.288542336Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-26T04:41:56.543 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.289106914Z level=info msg="Executing migration" id="create data_keys table" 2024-03-26T04:41:56.543 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.291949943Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T04:41:56.543 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.294677918Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T04:41:56.543 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.29900303Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T04:41:56.543 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.299933818Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T04:41:56.543 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.302657105Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T04:41:56.543 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.30687607Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.311119892Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.31412206Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.316845236Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.319486303Z level=info msg="Executing migration" id="create permission table" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.322256125Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.322720009Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.325663368Z level=info msg="Executing migration" id="create role table" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.328410484Z level=info msg="Executing migration" id="add column display_name" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.33093301Z level=info msg="Executing migration" id="add column group_name" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.331867228Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.334641436Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.337472595Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.340146783Z level=info msg="Executing migration" id="create team role table" 2024-03-26T04:41:56.544 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.34057443Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-26T04:41:56.545 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.343266846Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T04:41:56.545 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.346051862Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-26T04:41:56.545 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.348692328Z level=info msg="Executing migration" id="create user role table" 2024-03-26T04:41:56.545 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.349161907Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-26T04:41:56.545 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.351918541Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T04:41:56.545 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.354653639Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.357428608Z level=info msg="Executing migration" id="create builtin role table" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.357856363Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.360704812Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.363439491Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.364587785Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.365090567Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.365959538Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.366803788Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.367638981Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.368055115Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.368902942Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.369879557Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.370715591Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T04:41:56.546 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.371204843Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.372102927Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.372619085Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.373075801Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.373943144Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.374723467Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.375662223Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.376618272Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.377240158Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.378252428Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.379257758Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.379978907Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.380975247Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.381588492Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.382549687Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.383443764Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.384413937Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T04:41:56.547 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.384540133Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.384844188Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.384928179Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.385255146Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.387847405Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.388522417Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.390681615Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.39117455Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.393862929Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.396544371Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.399235445Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.399673206Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.402377759Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.405052476Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.407784096Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T04:41:56.548 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.410763569Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.413544374Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.416292936Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.419147042Z level=info msg="Executing migration" id="add share column" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.420187208Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.422879493Z level=info msg="Executing migration" id="create file table" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.423725397Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.424530412Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.42503733Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.425451957Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.426437521Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.427096755Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.427495418Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.428392611Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.42905201Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.430200176Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.431065484Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T04:41:56.549 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.431887733Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.43275271Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.433530586Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.434337911Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.435394875Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.436427012Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.437304738Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.446642929Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.447133534Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.448025703Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.451360853Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.452212346Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.452589336Z level=info msg="Executing migration" id="create folder table" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.453468274Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T04:41:56.550 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.454347752Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T04:41:56.551 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.455195305Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T04:41:56.551 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.45552081Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T04:41:56.551 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=migrator t=2024-03-26T04:41:56.456381184Z level=info msg="migrations completed" performed=485 skipped=0 duration=620.558309ms 2024-03-26T04:41:56.551 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=sqlstore t=2024-03-26T04:41:56.456810755Z level=info msg="Created default organization" 2024-03-26T04:41:56.551 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=plugin.loader t=2024-03-26T04:41:56.514605543Z level=info msg="Plugin registered" pluginID=input 2024-03-26T04:41:56.551 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=plugin.loader t=2024-03-26T04:41:56.542146232Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-26T04:41:56.551 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=plugin.loader t=2024-03-26T04:41:56.542165974Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-26T04:41:56.551 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=secrets t=2024-03-26T04:41:56.542196855Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T04:41:56.551 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=query_data t=2024-03-26T04:41:56.54435466Z level=info msg="Query Service initialization" 2024-03-26T04:41:56.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:56 smithi177 ceph-mon[27677]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:56.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:56 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:56.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:56 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:56.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:56 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:56.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:56 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:56.867 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:56 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=live.push_http t=2024-03-26T04:41:56.546782521Z level=info msg="Live Push Gateway initialization" 2024-03-26T04:41:57.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:57 smithi038 ceph-mon[25836]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:57.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:57 smithi038 ceph-mon[25836]: Deploying daemon node-exporter.a on smithi038 2024-03-26T04:41:57.332 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:41:57 smithi038 bash[50925]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T04:41:57.448 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-mon[27627]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:57.448 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-mon[27627]: Deploying daemon node-exporter.a on smithi038 2024-03-26T04:41:57.448 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=infra.usagestats.collector t=2024-03-26T04:41:57.187945747Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T04:41:57.449 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=provisioning.datasources t=2024-03-26T04:41:57.188903597Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T04:41:57.449 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=provisioning.datasources t=2024-03-26T04:41:57.199767628Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T04:41:57.449 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=provisioning.alerting t=2024-03-26T04:41:57.210585439Z level=info msg="starting to provision alerting" 2024-03-26T04:41:57.449 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=provisioning.alerting t=2024-03-26T04:41:57.2106046Z level=info msg="finished to provision alerting" 2024-03-26T04:41:57.449 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=grafanaStorageLogger t=2024-03-26T04:41:57.210758978Z level=info msg="storage starting" 2024-03-26T04:41:57.449 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=ngalert.state.manager t=2024-03-26T04:41:57.210803033Z level=info msg="Warming state cache for startup" 2024-03-26T04:41:57.449 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=http.server t=2024-03-26T04:41:57.212478629Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T04:41:57.449 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=ngalert.state.manager t=2024-03-26T04:41:57.216311645Z level=info msg="State cache has been initialized" states=0 duration=5.507776ms 2024-03-26T04:41:57.449 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=ngalert.multiorg.alertmanager t=2024-03-26T04:41:57.216374633Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T04:41:57.449 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=ticker t=2024-03-26T04:41:57.216385431Z level=info msg=starting first_tick=2024-03-26T04:42:00Z 2024-03-26T04:41:57.449 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:41:57 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a[38367]: logger=sqlstore.transactions t=2024-03-26T04:41:57.232619068Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T04:41:57.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:57 smithi177 ceph-mon[27677]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:57.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:57 smithi177 ceph-mon[27677]: Deploying daemon node-exporter.a on smithi038 2024-03-26T04:41:57.766 INFO:teuthology.orchestra.run.smithi038.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:41:57.766 INFO:teuthology.orchestra.run.smithi038.stdout:agent.smithi038 smithi038 running 5s ago 3m - - 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:agent.smithi114 smithi114 running 10s ago 3m - - 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:agent.smithi177 smithi177 running 8s ago 3m - - 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:alertmanager.a smithi038 *:9093,9094 running (16s) 5s ago 15s 17.8M - 0.25.0 c8568f914cd2 05ae4c96274e 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:grafana.a smithi114 *:3000 starting - - - - 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.x smithi114 *:8443,8765 running (2m) 10s ago 2m 449M - 19.0.0-1775-g5e70124b 5f21c9264136 5b35811f1de0 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.y smithi038 *:9283,8765 running (4m) 5s ago 4m 501M - 19.0.0-1775-g5e70124b 5f21c9264136 c2e9f563228d 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.z smithi177 *:8443,8765 running (2m) 8s ago 2m 449M - 19.0.0-1775-g5e70124b 5f21c9264136 f86597608c6f 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:mon.a smithi038 running (4m) 5s ago 5m 47.5M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 4de8559ee353 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:mon.b smithi114 running (3m) 10s ago 3m 35.6M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 11c8da4b99a3 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:mon.c smithi177 running (3m) 8s ago 3m 39.7M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 6cd6df0269c4 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 smithi038 running (2m) 5s ago 2m 52.2M 4096M 19.0.0-1775-g5e70124b 5f21c9264136 1876c5c0e475 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 smithi114 running (119s) 10s ago 119s 61.8M 4096M 19.0.0-1775-g5e70124b 5f21c9264136 352cc8548575 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 smithi177 running (95s) 8s ago 94s 63.4M 4096M 19.0.0-1775-g5e70124b 5f21c9264136 d8084d88db72 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:prometheus.a smithi114 *:9095 running (48s) 10s ago 48s 26.2M - 2.43.0 a07b618ecd1d bcee2c3113c0 2024-03-26T04:41:57.767 INFO:teuthology.orchestra.run.smithi038.stdout:rgw.foo.a smithi038 *:80 running (71s) 5s ago 70s 78.7M - 19.0.0-1775-g5e70124b 5f21c9264136 d6b074fa023e 2024-03-26T04:41:57.768 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:41:57 smithi038 bash[50925]: Getting image source signatures 2024-03-26T04:41:57.768 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:41:57 smithi038 bash[50925]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T04:41:57.768 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:41:57 smithi038 bash[50925]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T04:41:57.768 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:41:57 smithi038 bash[50925]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T04:41:59.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:58 smithi038 ceph-mon[25836]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:58 smithi038 ceph-mon[25836]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:59.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:41:58 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:59.122 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:58 smithi114 ceph-mon[27627]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:59.122 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:58 smithi114 ceph-mon[27627]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:59.122 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:41:58 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:59.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:58 smithi177 ceph-mon[27677]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:41:59.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:58 smithi177 ceph-mon[27677]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:59.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:41:58 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:41:59.338 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-26T04:41:59.580 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:41:59 smithi038 bash[50925]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T04:41:59.581 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:41:59 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: ::ffff:172.21.15.114 - - [26/Mar/2024:04:41:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:41:59.581 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:59.890 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:41:59 smithi038 bash[50925]: Writing manifest to image destination 2024-03-26T04:42:00.174 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:41:59 smithi038 podman[50925]: 2024-03-26 04:41:59.845953802 +0000 UTC m=+2.860900992 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T04:42:00.174 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:41:59 smithi038 podman[50925]: 2024-03-26 04:41:59.944922204 +0000 UTC m=+2.959869424 container create 51ab7484785446889e8a51362fe92dd73b8a33b4c5c08ec5c5f70887b69dcb9b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:00.941 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-mon[25836]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:00.941 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:00.942 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 podman[50925]: 2024-03-26 04:42:00.666205027 +0000 UTC m=+3.681152205 container init 51ab7484785446889e8a51362fe92dd73b8a33b4c5c08ec5c5f70887b69dcb9b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:00.942 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 podman[50925]: 2024-03-26 04:42:00.67220435 +0000 UTC m=+3.687151529 container start 51ab7484785446889e8a51362fe92dd73b8a33b4c5c08ec5c5f70887b69dcb9b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:00.942 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.676Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T04:42:00.942 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.676Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T04:42:00.942 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.676Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T04:42:00.942 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.676Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T04:42:00.942 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T04:42:00.942 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T04:42:00.943 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T04:42:00.944 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T04:42:00.945 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T04:42:00.945 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T04:42:00.945 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T04:42:00.945 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T04:42:00.945 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T04:42:00.945 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T04:42:00.945 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T04:42:00.945 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T04:42:00.945 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T04:42:00.946 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T04:42:00.946 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T04:42:00.946 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T04:42:00.946 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T04:42:00.946 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T04:42:00.946 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T04:42:00.946 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T04:42:00.946 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a[51118]: ts=2024-03-26T04:42:00.677Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T04:42:00.946 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 bash[50925]: 51ab7484785446889e8a51362fe92dd73b8a33b4c5c08ec5c5f70887b69dcb9b 2024-03-26T04:42:00.946 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:42:00 smithi038 systemd[1]: Started Ceph node-exporter.a for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:42:01.106 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:00 smithi114 ceph-mon[27627]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:01.106 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:00 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:01.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:00 smithi177 ceph-mon[27677]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:01.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:00 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:01.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:01 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:01.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:01 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:01.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:01 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:01.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:01 smithi038 ceph-mon[25836]: Deploying daemon node-exporter.b on smithi114 2024-03-26T04:42:02.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:01 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:02.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:01 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:02.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:01 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:02.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:01 smithi114 ceph-mon[27627]: Deploying daemon node-exporter.b on smithi114 2024-03-26T04:42:02.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:01 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:02.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:01 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:02.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:01 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:02.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:01 smithi177 ceph-mon[27677]: Deploying daemon node-exporter.b on smithi114 2024-03-26T04:42:02.882 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:02 smithi038 ceph-mon[25836]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:03.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:02 smithi114 ceph-mon[27627]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:03.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:02 smithi177 ceph-mon[27677]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:03.836 INFO:teuthology.orchestra.run.smithi038.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-26T04:42:03.837 INFO:teuthology.orchestra.run.smithi038.stdout:agent 3/3 16s ago 46s * 2024-03-26T04:42:03.837 INFO:teuthology.orchestra.run.smithi038.stdout:alertmanager ?:9093,9094 1/1 11s ago 71s smithi038=a;count:1 2024-03-26T04:42:03.837 INFO:teuthology.orchestra.run.smithi038.stdout:grafana ?:3000 0/1 - 69s smithi114=a;count:1 2024-03-26T04:42:03.837 INFO:teuthology.orchestra.run.smithi038.stdout:mgr 3/3 16s ago 2m smithi038=y;smithi114=x;smithi177=z;count:3 2024-03-26T04:42:03.837 INFO:teuthology.orchestra.run.smithi038.stdout:mon 3/3 16s ago 3m smithi038:172.21.15.38=a;smithi114:172.21.15.114=b;smithi177:172.21.15.177=c;count:3 2024-03-26T04:42:03.837 INFO:teuthology.orchestra.run.smithi038.stdout:node-exporter ?:9100 0/3 - 74s smithi038=a;smithi114=b;smithi177=c;count:3 2024-03-26T04:42:03.837 INFO:teuthology.orchestra.run.smithi038.stdout:osd 3 16s ago - 2024-03-26T04:42:03.837 INFO:teuthology.orchestra.run.smithi038.stdout:prometheus ?:9095 1/1 16s ago 76s smithi114=a;count:1 2024-03-26T04:42:03.837 INFO:teuthology.orchestra.run.smithi038.stdout:rgw.foo.a ?:80 1/1 11s ago 76s smithi038=foo.a;count:1 2024-03-26T04:42:04.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:03 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:04.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:03 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:04.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:03 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:04.561 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-26T04:42:05.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:04 smithi038 ceph-mon[25836]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:05.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:04 smithi038 ceph-mon[25836]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:05.132 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:42:05.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:04 smithi114 ceph-mon[27627]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:05.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:04 smithi114 ceph-mon[27627]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:05.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:04 smithi177 ceph-mon[27677]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:05.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:04 smithi177 ceph-mon[27677]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:06.645 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:06 smithi114 bash[39264]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T04:42:07.042 INFO:teuthology.orchestra.run.smithi038.stdout:HOST ADDR LABELS STATUS 2024-03-26T04:42:07.042 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 172.21.15.38 2024-03-26T04:42:07.042 INFO:teuthology.orchestra.run.smithi038.stdout:smithi114 172.21.15.114 2024-03-26T04:42:07.042 INFO:teuthology.orchestra.run.smithi038.stdout:smithi177 172.21.15.177 2024-03-26T04:42:07.042 INFO:teuthology.orchestra.run.smithi038.stdout:3 hosts in cluster 2024-03-26T04:42:07.372 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:06 smithi114 bash[39264]: Getting image source signatures 2024-03-26T04:42:07.372 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:06 smithi114 bash[39264]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T04:42:07.372 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:06 smithi114 bash[39264]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T04:42:07.372 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:06 smithi114 bash[39264]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T04:42:07.608 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:07 smithi038 ceph-mon[25836]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:07.608 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:07 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:07.608 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:07 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:07.608 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:07 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:07.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-26T04:42:07.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:07 smithi114 ceph-mon[27627]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:07.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:07 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:07.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:07 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:07.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:07 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:07.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:07 smithi177 ceph-mon[27677]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:07.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:07 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:07.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:07 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:07.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:07 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:07.907 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:42:08.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:08 smithi038 ceph-mon[25836]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:08.603 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:08 smithi177 ceph-mon[27677]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:08.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:08 smithi114 ceph-mon[27627]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:08.645 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:08 smithi114 bash[39264]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T04:42:08.645 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:08 smithi114 bash[39264]: Writing manifest to image destination 2024-03-26T04:42:09.146 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:08 smithi114 podman[39264]: 2024-03-26 04:42:08.73395989 +0000 UTC m=+2.228622821 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T04:42:09.146 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:08 smithi114 podman[39264]: 2024-03-26 04:42:08.849555486 +0000 UTC m=+2.344218448 container create a9c4f8137b39dd0429c4dddd02bffe821a1e2c6942c6ae3e4ca7c34e9f06b744 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:42:09.146 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 podman[39264]: 2024-03-26 04:42:09.051433059 +0000 UTC m=+2.546095979 container init a9c4f8137b39dd0429c4dddd02bffe821a1e2c6942c6ae3e4ca7c34e9f06b744 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:42:09.146 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 podman[39264]: 2024-03-26 04:42:09.056879879 +0000 UTC m=+2.551542800 container start a9c4f8137b39dd0429c4dddd02bffe821a1e2c6942c6ae3e4ca7c34e9f06b744 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:42:09.146 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.061Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T04:42:09.147 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.061Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T04:42:09.147 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.061Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T04:42:09.147 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.061Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T04:42:09.147 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.061Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T04:42:09.147 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.061Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T04:42:09.147 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T04:42:09.147 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T04:42:09.147 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T04:42:09.147 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T04:42:09.147 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T04:42:09.148 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T04:42:09.148 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T04:42:09.148 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T04:42:09.148 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T04:42:09.148 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T04:42:09.148 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T04:42:09.148 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T04:42:09.148 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T04:42:09.148 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T04:42:09.148 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T04:42:09.148 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T04:42:09.148 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T04:42:09.148 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T04:42:09.149 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T04:42:09.150 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T04:42:09.150 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T04:42:09.150 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T04:42:09.150 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T04:42:09.150 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T04:42:09.150 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T04:42:09.150 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T04:42:09.150 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T04:42:09.150 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T04:42:09.150 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T04:42:09.150 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T04:42:09.150 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T04:42:09.150 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b[39313]: ts=2024-03-26T04:42:09.062Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T04:42:09.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:09 smithi038 ceph-mon[25836]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:09.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:09 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:09.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:09 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:09.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:09 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:09.581 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:09 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: ::ffff:172.21.15.114 - - [26/Mar/2024:04:42:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:42:09.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-mon[27627]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:09.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:09.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:09.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:09 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:09.646 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 bash[39264]: a9c4f8137b39dd0429c4dddd02bffe821a1e2c6942c6ae3e4ca7c34e9f06b744 2024-03-26T04:42:09.646 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:42:09 smithi114 systemd[1]: Started Ceph node-exporter.b for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:42:09.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:09 smithi177 ceph-mon[27677]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:09.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:09 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:09.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:09 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:09.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:09 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-26T04:42:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN 3726G No 17s ago Has a FileSystem, LVM detected 2024-03-26T04:42:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 /dev/nvme1n1 ssd Linux_42987afa66f2869e95ec 89.4G Yes 17s ago 2024-03-26T04:42:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 /dev/nvme2n1 ssd Linux_785c9bc245ec3afce867 89.4G Yes 17s ago 2024-03-26T04:42:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 /dev/nvme3n1 ssd Linux_6da9fb742065fdcf1a12 89.4G Yes 17s ago 2024-03-26T04:42:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 /dev/nvme4n1 ssd Linux_bd03baa47c07a6a493f6 89.4G No 17s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-26T04:42:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W4J33D 931G No 17s ago Has partitions 2024-03-26T04:42:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:smithi114 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400WB400BGN 372G No 4s ago Has a FileSystem, LVM detected 2024-03-26T04:42:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:smithi114 /dev/nvme1n1 ssd Linux_e51e45fb85e9164a05a8 89.4G Yes 4s ago 2024-03-26T04:42:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:smithi114 /dev/nvme2n1 ssd Linux_9be6cd64173c051bffe5 89.4G Yes 4s ago 2024-03-26T04:42:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:smithi114 /dev/nvme3n1 ssd Linux_8398787abebc4cfadfba 89.4G Yes 4s ago 2024-03-26T04:42:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:smithi114 /dev/nvme4n1 ssd Linux_7af3eda6431ec951488c 89.4G No 4s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-26T04:42:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:smithi114 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5QYVM 931G No 4s ago Has partitions 2024-03-26T04:42:10.456 INFO:teuthology.orchestra.run.smithi038.stdout:smithi177 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233007S400BGN 372G No 0s ago Has a FileSystem, LVM detected 2024-03-26T04:42:10.456 INFO:teuthology.orchestra.run.smithi038.stdout:smithi177 /dev/nvme1n1 ssd Linux_a130a442367705bc3944 89.4G Yes 0s ago 2024-03-26T04:42:10.456 INFO:teuthology.orchestra.run.smithi038.stdout:smithi177 /dev/nvme2n1 ssd Linux_51d01293df68df9b27cd 89.4G Yes 0s ago 2024-03-26T04:42:10.456 INFO:teuthology.orchestra.run.smithi038.stdout:smithi177 /dev/nvme3n1 ssd Linux_c14b989b13a97bae137c 89.4G Yes 0s ago 2024-03-26T04:42:10.456 INFO:teuthology.orchestra.run.smithi038.stdout:smithi177 /dev/nvme4n1 ssd Linux_f7b8c90c5e43314389d7 89.4G No 0s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-26T04:42:10.456 INFO:teuthology.orchestra.run.smithi038.stdout:smithi177 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XNXN 931G No 0s ago Has partitions 2024-03-26T04:42:10.518 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:10 smithi177 ceph-mon[27677]: Deploying daemon node-exporter.c on smithi177 2024-03-26T04:42:10.518 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:10 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:10.518 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:10 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:10.518 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:10 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:10.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:10 smithi114 ceph-mon[27627]: Deploying daemon node-exporter.c on smithi177 2024-03-26T04:42:10.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:10 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:10.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:10 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:10.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:10 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:10.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:10 smithi038 ceph-mon[25836]: Deploying daemon node-exporter.c on smithi177 2024-03-26T04:42:10.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:10 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:10.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:10 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:10.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:10 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:11.227 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-26T04:42:11.346 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:11 smithi177 bash[39824]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T04:42:11.561 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:42:11.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:11 smithi038 ceph-mon[25836]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:11.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:11 smithi114 ceph-mon[27627]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:11.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:11 smithi177 ceph-mon[27677]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:12.149 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:11 smithi177 bash[39824]: Getting image source signatures 2024-03-26T04:42:12.149 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:11 smithi177 bash[39824]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T04:42:12.149 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:11 smithi177 bash[39824]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T04:42:12.149 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:11 smithi177 bash[39824]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T04:42:12.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:12 smithi038 ceph-mon[25836]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:12.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:12 smithi114 ceph-mon[27627]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:12.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:12 smithi177 ceph-mon[27677]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:13.149 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:12 smithi177 bash[39824]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T04:42:13.149 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 bash[39824]: Writing manifest to image destination 2024-03-26T04:42:13.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-mon[27677]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:13.613 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.613 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.613 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.613 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.613 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.613 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.613 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 podman[39824]: 2024-03-26 04:42:13.26302742 +0000 UTC m=+2.070798782 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T04:42:13.614 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 podman[39824]: 2024-03-26 04:42:13.361688188 +0000 UTC m=+2.169459555 container create f6a3b7b4fd11f2a24991f7cb8dd84f93a74ca20aa8fe282af1b80ae6cf1268eb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-26T04:42:13.614 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 podman[39824]: 2024-03-26 04:42:13.613339283 +0000 UTC m=+2.421110647 container init f6a3b7b4fd11f2a24991f7cb8dd84f93a74ca20aa8fe282af1b80ae6cf1268eb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-26T04:42:13.689 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:13 smithi038 ceph-mon[25836]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:13.689 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:13 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.689 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:13 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.689 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:13 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.690 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:13 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.690 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:13 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.690 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:13 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.792 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:13 smithi114 ceph-mon[27627]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:13.792 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:13 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.792 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:13 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.792 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:13 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.793 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:13 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.793 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:13 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.793 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:13 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:13.900 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 podman[39824]: 2024-03-26 04:42:13.618615985 +0000 UTC m=+2.426387349 container start f6a3b7b4fd11f2a24991f7cb8dd84f93a74ca20aa8fe282af1b80ae6cf1268eb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-26T04:42:13.900 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.622Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T04:42:13.900 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.622Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T04:42:13.900 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T04:42:13.900 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T04:42:13.900 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T04:42:13.900 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T04:42:13.900 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T04:42:13.900 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T04:42:13.900 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T04:42:13.900 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T04:42:13.901 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T04:42:13.901 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T04:42:13.901 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T04:42:13.901 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T04:42:13.901 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T04:42:13.901 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T04:42:13.901 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T04:42:13.901 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T04:42:13.901 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T04:42:13.902 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T04:42:13.902 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T04:42:13.902 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T04:42:13.902 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.623Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T04:42:13.902 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T04:42:13.902 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T04:42:13.902 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T04:42:13.902 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T04:42:13.902 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T04:42:13.902 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T04:42:13.902 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T04:42:13.902 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T04:42:13.902 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T04:42:13.903 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T04:42:13.903 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T04:42:13.903 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T04:42:13.903 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T04:42:13.903 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T04:42:13.903 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T04:42:13.903 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T04:42:13.903 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T04:42:13.903 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T04:42:13.903 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T04:42:13.903 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T04:42:13.903 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T04:42:13.903 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T04:42:13.904 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T04:42:13.904 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T04:42:13.904 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T04:42:13.904 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T04:42:13.904 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T04:42:13.904 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T04:42:13.904 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T04:42:13.904 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c[39886]: ts=2024-03-26T04:42:13.624Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T04:42:13.904 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 bash[39824]: f6a3b7b4fd11f2a24991f7cb8dd84f93a74ca20aa8fe282af1b80ae6cf1268eb 2024-03-26T04:42:13.904 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:42:13 smithi177 systemd[1]: Started Ceph node-exporter.c for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:42:14.200 INFO:teuthology.orchestra.run.smithi038.stdout:service_type: agent 2024-03-26T04:42:14.200 INFO:teuthology.orchestra.run.smithi038.stdout:service_name: agent 2024-03-26T04:42:14.200 INFO:teuthology.orchestra.run.smithi038.stdout:placement: 2024-03-26T04:42:14.200 INFO:teuthology.orchestra.run.smithi038.stdout: host_pattern: '*' 2024-03-26T04:42:14.200 INFO:teuthology.orchestra.run.smithi038.stdout:status: 2024-03-26T04:42:14.200 INFO:teuthology.orchestra.run.smithi038.stdout: created: '2024-03-26T04:41:17.777435Z' 2024-03-26T04:42:14.200 INFO:teuthology.orchestra.run.smithi038.stdout: last_refresh: '2024-03-26T04:42:06.367442Z' 2024-03-26T04:42:14.200 INFO:teuthology.orchestra.run.smithi038.stdout: running: 3 2024-03-26T04:42:14.200 INFO:teuthology.orchestra.run.smithi038.stdout: size: 3 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout:events: 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout:- 2024-03-26T04:41:19.262675Z service:agent [INFO] "service was created" 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout:--- 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout:service_type: alertmanager 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout:service_name: alertmanager 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout:placement: 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout: count: 1 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout: hosts: 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout: - smithi038=a 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout:status: 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout: created: '2024-03-26T04:40:52.154187Z' 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout: last_refresh: '2024-03-26T04:42:13.006038Z' 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout: ports: 2024-03-26T04:42:14.201 INFO:teuthology.orchestra.run.smithi038.stdout: - 9093 2024-03-26T04:42:14.202 INFO:teuthology.orchestra.run.smithi038.stdout: - 9094 2024-03-26T04:42:14.202 INFO:teuthology.orchestra.run.smithi038.stdout: running: 1 2024-03-26T04:42:14.202 INFO:teuthology.orchestra.run.smithi038.stdout: size: 1 2024-03-26T04:42:14.202 INFO:teuthology.orchestra.run.smithi038.stdout:events: 2024-03-26T04:42:14.202 INFO:teuthology.orchestra.run.smithi038.stdout:- 2024-03-26T04:41:42.243137Z service:alertmanager [INFO] "service was created" 2024-03-26T04:42:14.202 INFO:teuthology.orchestra.run.smithi038.stdout:--- 2024-03-26T04:42:14.202 INFO:teuthology.orchestra.run.smithi038.stdout:service_type: grafana 2024-03-26T04:42:14.202 INFO:teuthology.orchestra.run.smithi038.stdout:service_name: grafana 2024-03-26T04:42:14.202 INFO:teuthology.orchestra.run.smithi038.stdout:placement: 2024-03-26T04:42:14.202 INFO:teuthology.orchestra.run.smithi038.stdout: count: 1 2024-03-26T04:42:14.202 INFO:teuthology.orchestra.run.smithi038.stdout: hosts: 2024-03-26T04:42:14.202 INFO:teuthology.orchestra.run.smithi038.stdout: - smithi114=a 2024-03-26T04:42:14.202 INFO:teuthology.orchestra.run.smithi038.stdout:spec: 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout: anonymous_access: true 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout: protocol: https 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout:status: 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout: created: '2024-03-26T04:40:54.773990Z' 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout: last_refresh: '2024-03-26T04:42:06.367930Z' 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout: ports: 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout: - 3000 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout: running: 1 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout: size: 1 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout:events: 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout:- 2024-03-26T04:41:55.847825Z service:grafana [INFO] "service was created" 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout:--- 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout:service_type: mgr 2024-03-26T04:42:14.203 INFO:teuthology.orchestra.run.smithi038.stdout:service_name: mgr 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout:placement: 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout: count: 3 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout: hosts: 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout: - smithi038=y 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout: - smithi114=x 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout: - smithi177=z 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout:status: 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout: created: '2024-03-26T04:39:07.728095Z' 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout: last_refresh: '2024-03-26T04:42:06.367653Z' 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout: running: 3 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout: size: 3 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout:--- 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout:service_type: mon 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout:service_name: mon 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout:placement: 2024-03-26T04:42:14.204 INFO:teuthology.orchestra.run.smithi038.stdout: count: 3 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout: hosts: 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout: - smithi038:172.21.15.38=a 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout: - smithi114:172.21.15.114=b 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout: - smithi177:172.21.15.177=c 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout:status: 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout: created: '2024-03-26T04:38:41.122130Z' 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout: last_refresh: '2024-03-26T04:42:06.367558Z' 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout: running: 3 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout: size: 3 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout:--- 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout:service_type: node-exporter 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout:service_name: node-exporter 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout:placement: 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout: count: 3 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout: hosts: 2024-03-26T04:42:14.205 INFO:teuthology.orchestra.run.smithi038.stdout: - smithi038=a 2024-03-26T04:42:14.206 INFO:teuthology.orchestra.run.smithi038.stdout: - smithi114=b 2024-03-26T04:42:14.206 INFO:teuthology.orchestra.run.smithi038.stdout: - smithi177=c 2024-03-26T04:42:14.206 INFO:teuthology.orchestra.run.smithi038.stdout:status: 2024-03-26T04:42:14.206 INFO:teuthology.orchestra.run.smithi038.stdout: created: '2024-03-26T04:40:49.359585Z' 2024-03-26T04:42:14.206 INFO:teuthology.orchestra.run.smithi038.stdout: ports: 2024-03-26T04:42:14.206 INFO:teuthology.orchestra.run.smithi038.stdout: - 9100 2024-03-26T04:42:14.206 INFO:teuthology.orchestra.run.smithi038.stdout: running: 1 2024-03-26T04:42:14.206 INFO:teuthology.orchestra.run.smithi038.stdout: size: 3 2024-03-26T04:42:14.206 INFO:teuthology.orchestra.run.smithi038.stdout:events: 2024-03-26T04:42:14.206 INFO:teuthology.orchestra.run.smithi038.stdout:- 2024-03-26T04:42:13.790727Z service:node-exporter [INFO] "service was created" 2024-03-26T04:42:14.206 INFO:teuthology.orchestra.run.smithi038.stdout:--- 2024-03-26T04:42:14.206 INFO:teuthology.orchestra.run.smithi038.stdout:service_type: osd 2024-03-26T04:42:14.206 INFO:teuthology.orchestra.run.smithi038.stdout:service_name: osd 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout:unmanaged: true 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout:spec: 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout: filter_logic: AND 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout: objectstore: bluestore 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout:status: 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout: container_image_id: 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout: last_refresh: '2024-03-26T04:42:06.367742Z' 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout: running: 3 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout: size: 3 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout:--- 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout:service_type: prometheus 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout:service_name: prometheus 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout:placement: 2024-03-26T04:42:14.207 INFO:teuthology.orchestra.run.smithi038.stdout: count: 1 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout: hosts: 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout: - smithi114=a 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout:status: 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout: created: '2024-03-26T04:40:46.954716Z' 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout: last_refresh: '2024-03-26T04:42:06.367843Z' 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout: ports: 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout: - 9095 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout: running: 1 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout: size: 1 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout:events: 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout:- 2024-03-26T04:42:13.798480Z service:prometheus [INFO] "service was created" 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout:--- 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout:service_type: rgw 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout:service_id: foo.a 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout:service_name: rgw.foo.a 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout:placement: 2024-03-26T04:42:14.208 INFO:teuthology.orchestra.run.smithi038.stdout: count: 1 2024-03-26T04:42:14.209 INFO:teuthology.orchestra.run.smithi038.stdout: hosts: 2024-03-26T04:42:14.209 INFO:teuthology.orchestra.run.smithi038.stdout: - smithi038=foo.a 2024-03-26T04:42:14.209 INFO:teuthology.orchestra.run.smithi038.stdout:status: 2024-03-26T04:42:14.209 INFO:teuthology.orchestra.run.smithi038.stdout: created: '2024-03-26T04:40:47.294685Z' 2024-03-26T04:42:14.209 INFO:teuthology.orchestra.run.smithi038.stdout: last_refresh: '2024-03-26T04:42:13.005962Z' 2024-03-26T04:42:14.209 INFO:teuthology.orchestra.run.smithi038.stdout: ports: 2024-03-26T04:42:14.209 INFO:teuthology.orchestra.run.smithi038.stdout: - 80 2024-03-26T04:42:14.209 INFO:teuthology.orchestra.run.smithi038.stdout: running: 1 2024-03-26T04:42:14.209 INFO:teuthology.orchestra.run.smithi038.stdout: size: 1 2024-03-26T04:42:14.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:14.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:14.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:14.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:14.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:42:14.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-26T04:42:14.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:42:14.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:14 smithi038 ceph-mon[25836]: Reconfiguring daemon alertmanager.a on smithi038 2024-03-26T04:42:14.835 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-26T04:42:14.838 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:14.838 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.838 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.838 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.838 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.838 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.838 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:14.838 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:14.838 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:14.839 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.839 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:42:14.839 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-26T04:42:14.839 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:42:14.839 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:14 smithi114 ceph-mon[27627]: Reconfiguring daemon alertmanager.a on smithi038 2024-03-26T04:42:14.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:14.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:14.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:14.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:14.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:14.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T04:42:14.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-26T04:42:14.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:42:14.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:14 smithi177 ceph-mon[27677]: Reconfiguring daemon alertmanager.a on smithi038 2024-03-26T04:42:16.879 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:42:16.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:16 smithi114 ceph-mon[27627]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:16.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:16 smithi177 ceph-mon[27677]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:16 smithi038 ceph-mon[25836]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:18.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.331 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:18 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:18.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:18 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:18.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:18.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:18 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:19 smithi177 ceph-mon[27677]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:19 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:19 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:19.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:19 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:19.456 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:19 smithi114 ceph-mon[27627]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:19.456 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:19 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:19.456 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:19 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:19.456 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:19 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:19.580 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:19 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: ::ffff:172.21.15.114 - - [26/Mar/2024:04:42:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:42:19.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:19 smithi038 ceph-mon[25836]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:19.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:19 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:19.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:19 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:19.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:19 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:19.963 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:19 smithi038 systemd[1]: Stopping Ceph alertmanager.a for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:42:20.453 INFO:teuthology.orchestra.run.smithi038.stdout:osd 3 3s ago - 2024-03-26T04:42:20.816 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:20 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[49188]: ts=2024-03-26T04:42:20.350Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:42:20.816 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:20 smithi038 podman[53187]: 2024-03-26 04:42:20.371239193 +0000 UTC m=+0.114299592 container died 05ae4c96274e89ad70def610be7231ad915b711299e4e30a662471f6c32b3d7e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:21.081 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:20 smithi038 podman[53187]: 2024-03-26 04:42:20.924195368 +0000 UTC m=+0.667255760 container cleanup 05ae4c96274e89ad70def610be7231ad915b711299e4e30a662471f6c32b3d7e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:21.081 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:20 smithi038 bash[53187]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T04:42:21.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:20 smithi038 ceph-mon[25836]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:21.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:20 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:21.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:20 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:21.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:20 smithi038 ceph-mon[25836]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:21.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:20 smithi038 ceph-mon[25836]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:21.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:20 smithi114 ceph-mon[27627]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:21.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:20 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:21.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:20 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:21.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:20 smithi114 ceph-mon[27627]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:21.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:20 smithi114 ceph-mon[27627]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:21.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:20 smithi177 ceph-mon[27677]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:42:21.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:20 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:21.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:20 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:21.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:20 smithi177 ceph-mon[27677]: from='mgr.14442 172.21.15.38:0/3289442533' entity='mgr.y' 2024-03-26T04:42:21.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:20 smithi177 ceph-mon[27677]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:21.385 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:21 smithi038 podman[53203]: 2024-03-26 04:42:21.125433337 +0000 UTC m=+0.757195991 container remove 05ae4c96274e89ad70def610be7231ad915b711299e4e30a662471f6c32b3d7e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:21.386 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:21 smithi038 podman[53203]: 2024-03-26 04:42:21.201130924 +0000 UTC m=+0.832893536 volume remove 35864cca2bcd6f285f03bbe7f2181cbdd159184a4819d163ee9a4a9553f60f48 2024-03-26T04:42:21.658 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T04:42:21.669 INFO:tasks.cephadm:Teardown begin 2024-03-26T04:42:21.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:21.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:21.738 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:21.774 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T04:42:21.774 DEBUG:teuthology.orchestra.run.smithi038:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:42:21.789 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:42:21.805 DEBUG:teuthology.orchestra.run.smithi177:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:42:21.830 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T04:42:21.830 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T04:42:21.830 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.a 2024-03-26T04:42:21.916 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:21 smithi038 systemd[1]: Stopping Ceph mon.a for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:42:22.177 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:21 smithi038 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T04:42:22.177 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:21 smithi038 systemd[1]: Stopped Ceph alertmanager.a for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:42:22.177 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:21 smithi038 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 1.008s CPU time. 2024-03-26T04:42:22.177 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:21 smithi038 systemd[1]: Starting Ceph alertmanager.a for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:42:22.177 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:22 smithi038 podman[53409]: 2024-03-26 04:42:22.056199044 +0000 UTC m=+0.021251105 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T04:42:22.473 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:22 smithi038 podman[53409]: 2024-03-26 04:42:22.289027516 +0000 UTC m=+0.254079570 volume create 67da637bd5fd39dbd8c87d0d92c9492e9a153d9c88539e1c9f4c5b9eaf265814 2024-03-26T04:42:22.474 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:22 smithi038 podman[53409]: 2024-03-26 04:42:22.397869022 +0000 UTC m=+0.362921074 container create 0af118b0f7e713ad8463afa606843fffb195b29359325b624c2c28f7f8db4ee8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:22.474 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:22 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a[25813]: 2024-03-26T04:42:22.176+0000 7fd5a3ffe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:42:22.474 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:22 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a[25813]: 2024-03-26T04:42:22.176+0000 7fd5a3ffe700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T04:42:22.474 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:22 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:42:22] ENGINE Bus STOPPING 2024-03-26T04:42:22.474 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:22 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:42:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:42:22.474 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:22 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:42:22] ENGINE Bus STOPPED 2024-03-26T04:42:22.474 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:22 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:42:22] ENGINE Bus STARTING 2024-03-26T04:42:22.475 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:22 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:42:22] ENGINE Serving on http://:::9283 2024-03-26T04:42:22.475 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:22 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: [26/Mar/2024:04:42:22] ENGINE Bus STARTED 2024-03-26T04:42:22.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:22 smithi038 podman[53406]: 2024-03-26 04:42:22.474152726 +0000 UTC m=+0.442116857 container died 4de8559ee353a5d7537d40bf68ffc26d4a3959a6780fe3253b9686beda204f7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:42:23.009 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:22 smithi038 podman[53406]: 2024-03-26 04:42:22.825738691 +0000 UTC m=+0.793702811 container cleanup 4de8559ee353a5d7537d40bf68ffc26d4a3959a6780fe3253b9686beda204f7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux ) 2024-03-26T04:42:23.009 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:22 smithi038 bash[53406]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a 2024-03-26T04:42:23.311 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 04:42:23 smithi038 podman[53453]: 2024-03-26 04:42:23.010172782 +0000 UTC m=+0.564697500 container remove 4de8559ee353a5d7537d40bf68ffc26d4a3959a6780fe3253b9686beda204f7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T04:42:23.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T04:42:23.400 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:42:23.401 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T04:42:23.402 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-26T04:42:23.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.b 2024-03-26T04:42:23.581 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:23 smithi038 podman[53409]: 2024-03-26 04:42:23.312275405 +0000 UTC m=+1.277327457 container init 0af118b0f7e713ad8463afa606843fffb195b29359325b624c2c28f7f8db4ee8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:23.581 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:23 smithi038 podman[53409]: 2024-03-26 04:42:23.318518999 +0000 UTC m=+1.283571051 container start 0af118b0f7e713ad8463afa606843fffb195b29359325b624c2c28f7f8db4ee8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:23.581 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:23 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[53490]: ts=2024-03-26T04:42:23.331Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T04:42:23.581 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:23 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[53490]: ts=2024-03-26T04:42:23.331Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T04:42:23.581 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:23 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[53490]: ts=2024-03-26T04:42:23.333Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.38 port=9094 2024-03-26T04:42:23.581 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:23 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[53490]: ts=2024-03-26T04:42:23.333Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T04:42:23.582 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:23 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[53490]: ts=2024-03-26T04:42:23.358Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:42:23.582 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:23 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[53490]: ts=2024-03-26T04:42:23.359Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:42:23.582 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:23 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[53490]: ts=2024-03-26T04:42:23.362Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T04:42:23.582 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:23 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[53490]: ts=2024-03-26T04:42:23.362Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T04:42:23.582 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:23 smithi038 bash[53409]: 0af118b0f7e713ad8463afa606843fffb195b29359325b624c2c28f7f8db4ee8 2024-03-26T04:42:23.582 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:23 smithi038 systemd[1]: Started Ceph alertmanager.a for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:42:23.701 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:23 smithi114 systemd[1]: Stopping Ceph mon.b for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:42:23.957 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:23 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-b[27604]: 2024-03-26T04:42:23.699+0000 7fe69e308700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:42:23.957 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:23 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-b[27604]: 2024-03-26T04:42:23.699+0000 7fe69e308700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T04:42:24.257 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:23 smithi114 podman[40210]: 2024-03-26 04:42:23.957858324 +0000 UTC m=+0.363076721 container died 11c8da4b99a3c6a4dfe5c834f881bbb8b331d324f41045be8221618dd2f298b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T04:42:24.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:24 smithi114 podman[40210]: 2024-03-26 04:42:24.340585547 +0000 UTC m=+0.745803939 container cleanup 11c8da4b99a3c6a4dfe5c834f881bbb8b331d324f41045be8221618dd2f298b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-b, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:42:24.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:24 smithi114 bash[40210]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-b 2024-03-26T04:42:24.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 26 04:42:24 smithi114 podman[40229]: 2024-03-26 04:42:24.491422179 +0000 UTC m=+0.532996881 container remove 11c8da4b99a3c6a4dfe5c834f881bbb8b331d324f41045be8221618dd2f298b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-b, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-26T04:42:24.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T04:42:24.760 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:42:24.760 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-26T04:42:24.760 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-26T04:42:24.760 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.c 2024-03-26T04:42:25.071 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:24 smithi177 systemd[1]: Stopping Ceph mon.c for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:42:25.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:25 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-c[27641]: 2024-03-26T04:42:25.070+0000 7f9f451f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:42:25.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:25 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-c[27641]: 2024-03-26T04:42:25.070+0000 7f9f451f5700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T04:42:25.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:25 smithi177 podman[40594]: 2024-03-26 04:42:25.29704149 +0000 UTC m=+0.346305215 container died 6cd6df0269c42a0f9ec3c91c1f29f326e12df7ec978d6d68ff4f2013b2644908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-c, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T04:42:25.831 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:25 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[53490]: ts=2024-03-26T04:42:25.334Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000406124s 2024-03-26T04:42:25.894 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:25 smithi177 podman[40594]: 2024-03-26 04:42:25.660756006 +0000 UTC m=+0.710019733 container cleanup 6cd6df0269c42a0f9ec3c91c1f29f326e12df7ec978d6d68ff4f2013b2644908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T04:42:25.894 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 26 04:42:25 smithi177 bash[40594]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mon-c 2024-03-26T04:42:26.014 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T04:42:26.060 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:42:26.060 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-26T04:42:26.060 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T04:42:26.060 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.y 2024-03-26T04:42:26.366 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:26 smithi038 systemd[1]: Stopping Ceph mgr.y for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:42:26.697 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:26 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:42:26.365+0000 7fb70b77d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:42:26.698 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:26 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y[26091]: 2024-03-26T04:42:26.365+0000 7fb70b77d700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-26T04:42:26.698 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:26 smithi038 podman[53746]: 2024-03-26 04:42:26.397336603 +0000 UTC m=+0.140094828 container died c2e9f563228dd6f6a120cc8a70fb5e4c1c45c2b94b8769ed79b0e45bff066608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-26T04:42:27.081 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:26 smithi038 podman[53746]: 2024-03-26 04:42:26.880446947 +0000 UTC m=+0.623205211 container cleanup c2e9f563228dd6f6a120cc8a70fb5e4c1c45c2b94b8769ed79b0e45bff066608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T04:42:27.081 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:26 smithi038 bash[53746]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y 2024-03-26T04:42:27.307 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T04:42:27.331 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:27 smithi038 podman[53765]: 2024-03-26 04:42:27.082019417 +0000 UTC m=+0.682196127 container remove c2e9f563228dd6f6a120cc8a70fb5e4c1c45c2b94b8769ed79b0e45bff066608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:42:27.331 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:27 smithi038 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.y.service: Deactivated successfully. 2024-03-26T04:42:27.331 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:27 smithi038 systemd[1]: Stopped Ceph mgr.y for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:42:27.331 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 26 04:42:27 smithi038 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.y.service: Consumed 1min 27.353s CPU time. 2024-03-26T04:42:27.346 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:42:27.346 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T04:42:27.347 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T04:42:27.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.x 2024-03-26T04:42:27.645 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:42:27 smithi114 systemd[1]: Stopping Ceph mgr.x for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:42:27.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:42:27 smithi114 podman[40344]: 2024-03-26 04:42:27.66762628 +0000 UTC m=+0.135124098 container died 5b35811f1de0cd99bd8d3f5759bc035cb35572bb0477c15c545490852de5bd83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.license=GPLv2) 2024-03-26T04:42:28.177 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:42:28 smithi114 podman[40344]: 2024-03-26 04:42:28.001814895 +0000 UTC m=+0.469312713 container cleanup 5b35811f1de0cd99bd8d3f5759bc035cb35572bb0477c15c545490852de5bd83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_CLEAN=True) 2024-03-26T04:42:28.177 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:42:28 smithi114 bash[40344]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x 2024-03-26T04:42:28.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T04:42:28.441 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:42:28 smithi114 podman[40363]: 2024-03-26 04:42:28.178039899 +0000 UTC m=+0.505911315 container remove 5b35811f1de0cd99bd8d3f5759bc035cb35572bb0477c15c545490852de5bd83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-26T04:42:28.442 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:42:28 smithi114 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:42:28.442 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:42:28 smithi114 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-26T04:42:28.442 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:42:28 smithi114 systemd[1]: Stopped Ceph mgr.x for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:42:28.442 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Mar 26 04:42:28 smithi114 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.x.service: Consumed 16.764s CPU time. 2024-03-26T04:42:28.453 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:42:28.454 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T04:42:28.454 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-26T04:42:28.454 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.z 2024-03-26T04:42:28.762 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:42:28 smithi177 systemd[1]: Stopping Ceph mgr.z for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:42:29.039 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:42:28 smithi177 podman[40726]: 2024-03-26 04:42:28.763331384 +0000 UTC m=+0.120146126 container died f86597608c6ffe5c7c4ff0463207868354f869613775768ff197fe1f9a2d97b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:42:29.389 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:42:29 smithi177 podman[40726]: 2024-03-26 04:42:29.130732307 +0000 UTC m=+0.487547045 container cleanup f86597608c6ffe5c7c4ff0463207868354f869613775768ff197fe1f9a2d97b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:42:29.389 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 26 04:42:29 smithi177 bash[40726]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-mgr-z 2024-03-26T04:42:29.547 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-26T04:42:29.583 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:42:29.584 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-26T04:42:29.584 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T04:42:29.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-56caa506-eb2a-11ee-95d0-87774f69a715@osd.0 2024-03-26T04:42:29.879 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:29 smithi038 systemd[1]: Stopping Ceph osd.0 for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:42:30.330 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:29 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-0[38143]: 2024-03-26T04:42:29.878+0000 7f4e4e3d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:42:30.331 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:29 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-0[38143]: 2024-03-26T04:42:29.878+0000 7f4e4e3d3700 -1 osd.0 31 *** Got signal Terminated *** 2024-03-26T04:42:30.331 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:29 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-0[38143]: 2024-03-26T04:42:29.878+0000 7f4e4e3d3700 -1 osd.0 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:42:33.581 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:42:33 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[53490]: ts=2024-03-26T04:42:33.336Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002732534s 2024-03-26T04:42:35.581 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:35 smithi038 podman[53879]: 2024-03-26 04:42:35.328894914 +0000 UTC m=+5.555128275 container died 1876c5c0e47561d8004e3225dadb8d04c2c525ad1c418d1d1472fc759aa6ac6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T04:42:36.061 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:35 smithi038 podman[53879]: 2024-03-26 04:42:35.818946326 +0000 UTC m=+6.045179692 container cleanup 1876c5c0e47561d8004e3225dadb8d04c2c525ad1c418d1d1472fc759aa6ac6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:42:36.061 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:35 smithi038 bash[53879]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-0 2024-03-26T04:42:36.331 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:36 smithi038 podman[54384]: 2024-03-26 04:42:36.061884659 +0000 UTC m=+0.729067168 container remove 1876c5c0e47561d8004e3225dadb8d04c2c525ad1c418d1d1472fc759aa6ac6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-0, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T04:42:36.648 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:36 smithi038 podman[54480]: 2024-03-26 04:42:36.257527821 +0000 UTC m=+0.017718805 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:42:36.648 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:36 smithi038 podman[54480]: 2024-03-26 04:42:36.371257302 +0000 UTC m=+0.131448276 container create 98de8bcab656b46c1152fd4b2bebfa43a9352378eb0aa20ea627c59ba60a087a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T04:42:36.648 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:36 smithi038 podman[54480]: 2024-03-26 04:42:36.648602669 +0000 UTC m=+0.408793641 container init 98de8bcab656b46c1152fd4b2bebfa43a9352378eb0aa20ea627c59ba60a087a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124) 2024-03-26T04:42:36.954 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:36 smithi038 podman[54480]: 2024-03-26 04:42:36.654550796 +0000 UTC m=+0.414741768 container start 98de8bcab656b46c1152fd4b2bebfa43a9352378eb0aa20ea627c59ba60a087a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-26T04:42:36.954 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:36 smithi038 podman[54480]: 2024-03-26 04:42:36.74019141 +0000 UTC m=+0.500382382 container attach 98de8bcab656b46c1152fd4b2bebfa43a9352378eb0aa20ea627c59ba60a087a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-26T04:42:37.242 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 04:42:36 smithi038 podman[54480]: 2024-03-26 04:42:36.955275091 +0000 UTC m=+0.715466063 container died 98de8bcab656b46c1152fd4b2bebfa43a9352378eb0aa20ea627c59ba60a087a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T04:42:37.641 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T04:42:37.685 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:42:37.685 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T04:42:37.685 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T04:42:37.685 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-56caa506-eb2a-11ee-95d0-87774f69a715@osd.1 2024-03-26T04:42:38.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:37 smithi114 systemd[1]: Stopping Ceph osd.1 for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:42:38.645 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:38 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1[32903]: 2024-03-26T04:42:38.477+0000 7f60d1bd8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:42:38.645 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:38 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1[32903]: 2024-03-26T04:42:38.477+0000 7f60d1bd8700 -1 osd.1 31 *** Got signal Terminated *** 2024-03-26T04:42:38.645 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:38 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1[32903]: 2024-03-26T04:42:38.477+0000 7f60d1bd8700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:42:44.146 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:43 smithi114 podman[40838]: 2024-03-26 04:42:43.860668845 +0000 UTC m=+5.472117506 container died 352cc85485757807aac83ddd3554d73ad06214cf88c68a2b6da10df4c2e84ac1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T04:42:44.460 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:44 smithi114 podman[40838]: 2024-03-26 04:42:44.218235098 +0000 UTC m=+5.829683751 container cleanup 352cc85485757807aac83ddd3554d73ad06214cf88c68a2b6da10df4c2e84ac1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T04:42:44.460 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:44 smithi114 bash[40838]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1 2024-03-26T04:42:44.752 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:44 smithi114 podman[41213]: 2024-03-26 04:42:44.460869112 +0000 UTC m=+0.593434816 container remove 352cc85485757807aac83ddd3554d73ad06214cf88c68a2b6da10df4c2e84ac1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:42:45.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:44 smithi114 podman[41313]: 2024-03-26 04:42:44.753243604 +0000 UTC m=+0.095303185 container create 1071780f031b5d58e77014953ad889a3002037fdab99f05befbae827e5c12b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T04:42:45.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:44 smithi114 podman[41313]: 2024-03-26 04:42:44.67582357 +0000 UTC m=+0.017883165 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:42:45.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:44 smithi114 podman[41313]: 2024-03-26 04:42:44.93084122 +0000 UTC m=+0.272900805 container init 1071780f031b5d58e77014953ad889a3002037fdab99f05befbae827e5c12b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T04:42:45.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:44 smithi114 podman[41313]: 2024-03-26 04:42:44.935859857 +0000 UTC m=+0.277919450 container start 1071780f031b5d58e77014953ad889a3002037fdab99f05befbae827e5c12b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-26T04:42:45.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:45 smithi114 podman[41313]: 2024-03-26 04:42:45.005505873 +0000 UTC m=+0.347565473 container attach 1071780f031b5d58e77014953ad889a3002037fdab99f05befbae827e5c12b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:42:45.395 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:45 smithi114 podman[41313]: 2024-03-26 04:42:45.223075572 +0000 UTC m=+0.565135158 container died 1071780f031b5d58e77014953ad889a3002037fdab99f05befbae827e5c12b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T04:42:45.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 26 04:42:45 smithi114 podman[41358]: 2024-03-26 04:42:45.750147964 +0000 UTC m=+0.522088902 container remove 1071780f031b5d58e77014953ad889a3002037fdab99f05befbae827e5c12b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T04:42:45.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T04:42:45.817 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:42:45.817 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T04:42:45.817 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T04:42:45.817 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-56caa506-eb2a-11ee-95d0-87774f69a715@osd.2 2024-03-26T04:42:46.119 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:45 smithi177 systemd[1]: Stopping Ceph osd.2 for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:42:46.399 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:46 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2[34117]: 2024-03-26T04:42:46.119+0000 7f4ecc38e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:42:46.399 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:46 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2[34117]: 2024-03-26T04:42:46.119+0000 7f4ecc38e700 -1 osd.2 31 *** Got signal Terminated *** 2024-03-26T04:42:46.399 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:46 smithi177 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2[34117]: 2024-03-26T04:42:46.119+0000 7f4ecc38e700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:42:51.782 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:51 smithi177 podman[41417]: 2024-03-26 04:42:51.48440547 +0000 UTC m=+5.474064233 container died d8084d88db722226a6479eef9a50946823ac976c8d83671f011e952b677c133e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-03-26T04:42:51.783 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:51 smithi177 podman[41417]: 2024-03-26 04:42:51.782549329 +0000 UTC m=+5.772208091 container cleanup d8084d88db722226a6479eef9a50946823ac976c8d83671f011e952b677c133e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-26T04:42:52.149 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:51 smithi177 bash[41417]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2 2024-03-26T04:42:52.149 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:52 smithi177 podman[41451]: 2024-03-26 04:42:52.008337859 +0000 UTC m=+0.518555677 container remove d8084d88db722226a6479eef9a50946823ac976c8d83671f011e952b677c133e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_CLEAN=True) 2024-03-26T04:42:52.618 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:52 smithi177 podman[41544]: 2024-03-26 04:42:52.219553935 +0000 UTC m=+0.017455117 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:42:52.619 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:52 smithi177 podman[41544]: 2024-03-26 04:42:52.333918679 +0000 UTC m=+0.131819848 container create 34993b797be2c22640dfff3d7bb56cbba62a439880b06d24b6925e79a6177f36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T04:42:52.619 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:52 smithi177 podman[41544]: 2024-03-26 04:42:52.519458875 +0000 UTC m=+0.317360043 container init 34993b797be2c22640dfff3d7bb56cbba62a439880b06d24b6925e79a6177f36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T04:42:52.619 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:52 smithi177 podman[41544]: 2024-03-26 04:42:52.525349585 +0000 UTC m=+0.323250754 container start 34993b797be2c22640dfff3d7bb56cbba62a439880b06d24b6925e79a6177f36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2-deactivate, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-26T04:42:52.619 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:52 smithi177 podman[41544]: 2024-03-26 04:42:52.619209567 +0000 UTC m=+0.417110736 container attach 34993b797be2c22640dfff3d7bb56cbba62a439880b06d24b6925e79a6177f36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-26T04:42:52.899 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:52 smithi177 podman[41544]: 2024-03-26 04:42:52.813723868 +0000 UTC m=+0.611625039 container died 34993b797be2c22640dfff3d7bb56cbba62a439880b06d24b6925e79a6177f36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-26T04:42:53.354 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T04:42:53.380 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:53 smithi177 podman[41590]: 2024-03-26 04:42:53.338453415 +0000 UTC m=+0.518953531 container remove 34993b797be2c22640dfff3d7bb56cbba62a439880b06d24b6925e79a6177f36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:42:53.381 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:53 smithi177 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-26T04:42:53.381 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:53 smithi177 systemd[1]: Stopped Ceph osd.2 for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:42:53.381 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 26 04:42:53 smithi177 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@osd.2.service: Consumed 4.281s CPU time. 2024-03-26T04:42:53.395 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:42:53.396 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T04:42:53.396 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-26T04:42:53.396 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-56caa506-eb2a-11ee-95d0-87774f69a715@rgw.foo.a 2024-03-26T04:42:53.830 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 26 04:42:53 smithi038 systemd[1]: Stopping Ceph rgw.foo.a for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:42:54.518 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 26 04:42:54 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-rgw-foo-a[41383]: 2024-03-26T04:42:54.330+0000 7f741a303700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:42:54.518 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 26 04:42:54 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-rgw-foo-a[41383]: 2024-03-26T04:42:54.330+0000 7f7428b80ac0 -1 shutting down 2024-03-26T04:43:04.638 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 26 04:43:04 smithi038 bash[55140]: time="2024-03-26T04:43:04Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-56caa506-eb2a-11ee-95d0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-26T04:43:04.638 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 26 04:43:04 smithi038 podman[55140]: 2024-03-26 04:43:04.362582395 +0000 UTC m=+10.412160470 container died d6b074fa023ed5bde8fa6921494e80af98efc6ef1ee3ccd4b20994f60d95dc2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-rgw-foo-a, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-26T04:43:05.081 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 26 04:43:04 smithi038 podman[55140]: 2024-03-26 04:43:04.845968004 +0000 UTC m=+10.895546121 container cleanup d6b074fa023ed5bde8fa6921494e80af98efc6ef1ee3ccd4b20994f60d95dc2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-rgw-foo-a, RELEASE=squid-5e70124, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:43:05.081 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 26 04:43:04 smithi038 bash[55140]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-rgw-foo-a 2024-03-26T04:43:05.211 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@rgw.foo.a.service' 2024-03-26T04:43:05.250 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:43:05.251 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-26T04:43:05.251 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T04:43:05.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-56caa506-eb2a-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T04:43:05.550 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 systemd[1]: Stopping Ceph prometheus.a for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:43:05.863 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:43:05.550Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:43:05.864 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:43:05.550Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T04:43:05.864 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:43:05.550Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T04:43:05.864 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:43:05.550Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T04:43:05.864 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:43:05.550Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T04:43:05.864 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:43:05.550Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T04:43:05.864 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:43:05.550Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T04:43:05.864 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:43:05.550Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T04:43:05.864 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:43:05.550Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T04:43:05.864 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:43:05.551Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T04:43:05.864 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:43:05.551Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T04:43:05.864 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a[35609]: ts=2024-03-26T04:43:05.551Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T04:43:05.864 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 podman[42119]: 2024-03-26 04:43:05.561324891 +0000 UTC m=+0.114901079 container died bcee2c3113c0455d43a7500d7be27ec435accd786f11e42c42ec15ddc4f0b8ed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:43:05.865 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 podman[42119]: 2024-03-26 04:43:05.86405632 +0000 UTC m=+0.417632494 container cleanup bcee2c3113c0455d43a7500d7be27ec435accd786f11e42c42ec15ddc4f0b8ed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:43:05.865 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:05 smithi114 bash[42119]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T04:43:06.145 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Mar 26 04:43:06 smithi114 podman[42138]: 2024-03-26 04:43:06.073502694 +0000 UTC m=+0.515567676 container remove bcee2c3113c0455d43a7500d7be27ec435accd786f11e42c42ec15ddc4f0b8ed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:43:06.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-56caa506-eb2a-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T04:43:06.239 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:43:06.240 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T04:43:06.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T04:43:06.436 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:43:07.791 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:43:07 smithi038 systemd[1]: Stopping Ceph alertmanager.a for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:43:07.791 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:43:07 smithi038 ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a[53490]: ts=2024-03-26T04:43:07.770Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:43:08.080 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:43:07 smithi038 podman[55875]: 2024-03-26 04:43:07.790976061 +0000 UTC m=+0.117537223 container died 0af118b0f7e713ad8463afa606843fffb195b29359325b624c2c28f7f8db4ee8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:43:08.509 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:43:08 smithi038 podman[55875]: 2024-03-26 04:43:08.259140499 +0000 UTC m=+0.585701694 container cleanup 0af118b0f7e713ad8463afa606843fffb195b29359325b624c2c28f7f8db4ee8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:43:08.509 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:43:08 smithi038 bash[55875]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T04:43:08.831 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:43:08 smithi038 podman[55894]: 2024-03-26 04:43:08.510389528 +0000 UTC m=+0.718872861 container remove 0af118b0f7e713ad8463afa606843fffb195b29359325b624c2c28f7f8db4ee8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:43:08.831 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:43:08 smithi038 podman[55894]: 2024-03-26 04:43:08.586088272 +0000 UTC m=+0.794571605 volume remove 67da637bd5fd39dbd8c87d0d92c9492e9a153d9c88539e1c9f4c5b9eaf265814 2024-03-26T04:43:08.831 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:43:08 smithi038 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T04:43:08.831 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 26 04:43:08 smithi038 systemd[1]: Stopped Ceph alertmanager.a for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:43:09.331 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:43:08 smithi038 systemd[1]: Stopping Ceph node-exporter.a for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:43:09.331 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:43:09 smithi038 podman[56012]: 2024-03-26 04:43:09.193722028 +0000 UTC m=+0.112344262 container died 51ab7484785446889e8a51362fe92dd73b8a33b4c5c08ec5c5f70887b69dcb9b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:43:09.830 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:43:09 smithi038 podman[56012]: 2024-03-26 04:43:09.655325315 +0000 UTC m=+0.573947552 container cleanup 51ab7484785446889e8a51362fe92dd73b8a33b4c5c08ec5c5f70887b69dcb9b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:43:09.831 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:43:09 smithi038 bash[56012]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T04:43:10.186 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:43:09 smithi038 podman[56031]: 2024-03-26 04:43:09.906205995 +0000 UTC m=+0.713593156 container remove 51ab7484785446889e8a51362fe92dd73b8a33b4c5c08ec5c5f70887b69dcb9b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:43:10.186 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:43:09 smithi038 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:43:10.186 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:43:10 smithi038 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T04:43:10.186 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:43:10 smithi038 systemd[1]: Stopped Ceph node-exporter.a for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:43:10.186 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 26 04:43:10 smithi038 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 1.793s CPU time. 2024-03-26T04:43:10.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T04:43:10.868 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:43:12.131 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:43:11 smithi114 systemd[1]: Stopping Ceph grafana.a for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:43:12.384 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:43:12 smithi114 podman[42504]: 2024-03-26 04:43:12.131488273 +0000 UTC m=+0.124776311 container died acfb516583bf849100cc55822035d7bf16b2441106fcbffdfd9e50fafdbbe6d4 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a, version=8.5, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, vcs-type=git, vendor=Red Hat, Inc., io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, io.openshift.expose-services=, name=ubi8, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-03-26T04:43:12.384 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:43:12 smithi114 podman[42504]: 2024-03-26 04:43:12.384561608 +0000 UTC m=+0.377849638 container cleanup acfb516583bf849100cc55822035d7bf16b2441106fcbffdfd9e50fafdbbe6d4 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a, vcs-type=git, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, release=239.1651231664, distribution-scope=public, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, version=8.5, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.openshift.expose-services=, io.openshift.tags=base rhel8) 2024-03-26T04:43:12.645 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:43:12 smithi114 bash[42504]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a 2024-03-26T04:43:12.645 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:43:12 smithi114 podman[42523]: 2024-03-26 04:43:12.602612688 +0000 UTC m=+0.473548367 container remove acfb516583bf849100cc55822035d7bf16b2441106fcbffdfd9e50fafdbbe6d4 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , name=ubi8, architecture=x86_64, com.redhat.component=ubi8-container, release=239.1651231664, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, vcs-type=git, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-03-26T04:43:12.645 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:43:12 smithi114 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:43:13.145 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:43:12 smithi114 systemd[1]: Stopping Ceph node-exporter.b for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:43:13.146 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:43:12 smithi114 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T04:43:13.146 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:43:12 smithi114 systemd[1]: Stopped Ceph grafana.a for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:43:13.146 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Mar 26 04:43:12 smithi114 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@grafana.a.service: Consumed 3.695s CPU time. 2024-03-26T04:43:13.455 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:43:13 smithi114 podman[42640]: 2024-03-26 04:43:13.169184552 +0000 UTC m=+0.103288526 container died a9c4f8137b39dd0429c4dddd02bffe821a1e2c6942c6ae3e4ca7c34e9f06b744 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:43:13.730 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:43:13 smithi114 podman[42640]: 2024-03-26 04:43:13.572024039 +0000 UTC m=+0.506128013 container cleanup a9c4f8137b39dd0429c4dddd02bffe821a1e2c6942c6ae3e4ca7c34e9f06b744 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:43:13.731 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:43:13 smithi114 bash[42640]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T04:43:14.088 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:43:13 smithi114 podman[42659]: 2024-03-26 04:43:13.731453399 +0000 UTC m=+0.563748460 container remove a9c4f8137b39dd0429c4dddd02bffe821a1e2c6942c6ae3e4ca7c34e9f06b744 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:43:14.088 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:43:13 smithi114 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:43:14.088 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:43:13 smithi114 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T04:43:14.088 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:43:13 smithi114 systemd[1]: Stopped Ceph node-exporter.b for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:43:14.088 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Mar 26 04:43:13 smithi114 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 1.737s CPU time. 2024-03-26T04:43:14.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T04:43:14.760 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:43:15.840 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:43:15 smithi177 systemd[1]: Stopping Ceph node-exporter.c for 56caa506-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:43:16.118 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:43:15 smithi177 podman[42369]: 2024-03-26 04:43:15.840535516 +0000 UTC m=+0.116496701 container died f6a3b7b4fd11f2a24991f7cb8dd84f93a74ca20aa8fe282af1b80ae6cf1268eb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-26T04:43:16.399 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:43:16 smithi177 podman[42369]: 2024-03-26 04:43:16.227259971 +0000 UTC m=+0.503221166 container cleanup f6a3b7b4fd11f2a24991f7cb8dd84f93a74ca20aa8fe282af1b80ae6cf1268eb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-26T04:43:16.399 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:43:16 smithi177 bash[42369]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c 2024-03-26T04:43:16.738 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:43:16 smithi177 podman[42388]: 2024-03-26 04:43:16.444849868 +0000 UTC m=+0.605507233 container remove f6a3b7b4fd11f2a24991f7cb8dd84f93a74ca20aa8fe282af1b80ae6cf1268eb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-56caa506-eb2a-11ee-95d0-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-26T04:43:16.738 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:43:16 smithi177 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@node-exporter.c.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:43:16.738 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:43:16 smithi177 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@node-exporter.c.service: Failed with result 'exit-code'. 2024-03-26T04:43:16.738 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:43:16 smithi177 systemd[1]: Stopped Ceph node-exporter.c for 56caa506-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:43:16.739 INFO:journalctl@ceph.node-exporter.c.smithi177.stdout:Mar 26 04:43:16 smithi177 systemd[1]: ceph-56caa506-eb2a-11ee-95d0-87774f69a715@node-exporter.c.service: Consumed 1.813s CPU time. 2024-03-26T04:43:17.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:43:17.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:43:17.295 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:43:17.325 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T04:43:17.326 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028/remote/smithi038/crash 2024-03-26T04:43:17.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/crash -- . 2024-03-26T04:43:17.372 INFO:teuthology.orchestra.run.smithi038.stderr:tar: /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T04:43:17.372 INFO:teuthology.orchestra.run.smithi038.stderr:tar: Error is not recoverable: exiting now 2024-03-26T04:43:17.374 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028/remote/smithi114/crash 2024-03-26T04:43:17.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/crash -- . 2024-03-26T04:43:17.421 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T04:43:17.421 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now 2024-03-26T04:43:17.423 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028/remote/smithi177/crash 2024-03-26T04:43:17.424 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/crash -- . 2024-03-26T04:43:17.461 INFO:teuthology.orchestra.run.smithi177.stderr:tar: /var/lib/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T04:43:17.462 INFO:teuthology.orchestra.run.smithi177.stderr:tar: Error is not recoverable: exiting now 2024-03-26T04:43:17.463 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T04:43:17.463 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T04:43:17.495 INFO:teuthology.orchestra.run.smithi038.stdout:2024-03-26T04:38:59.972242+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:43:17.495 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T04:43:17.495 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[SEC\]' /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T04:43:17.565 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]' /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T04:43:17.636 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[WRN\]' /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T04:43:17.704 INFO:teuthology.orchestra.run.smithi038.stdout:2024-03-26T04:38:59.972242+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:43:17.704 INFO:tasks.cephadm:Compressing logs... 2024-03-26T04:43:17.705 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T04:43:17.747 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T04:43:17.749 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T04:43:17.772 INFO:teuthology.orchestra.run.smithi038.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T04:43:17.772 INFO:teuthology.orchestra.run.smithi038.stderr:: No such file or directory 2024-03-26T04:43:17.772 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T04:43:17.772 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/cephadm.log: /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T04:43:17.774 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-26T04:43:17.775 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T04:43:17.775 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T04:43:17.775 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mgr.y.log: 57.6% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-26T04:43:17.775 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T04:43:17.775 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.log 2024-03-26T04:43:17.775 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T04:43:17.775 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.log: 82.6% 89.9% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T04:43:17.775 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T04:43:17.775 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-volume.log: 85.1% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T04:43:17.776 INFO:teuthology.orchestra.run.smithi038.stderr: 83.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T04:43:17.777 INFO:teuthology.orchestra.run.smithi177.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T04:43:17.777 INFO:teuthology.orchestra.run.smithi177.stderr:: No such file or directory 2024-03-26T04:43:17.778 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T04:43:17.778 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-26T04:43:17.778 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T04:43:17.779 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T04:43:17.779 INFO:teuthology.orchestra.run.smithi114.stderr:: No such file or directory 2024-03-26T04:43:17.779 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T04:43:17.779 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T04:43:17.779 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T04:43:17.779 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T04:43:17.779 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.log 2024-03-26T04:43:17.779 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T04:43:17.779 INFO:teuthology.orchestra.run.smithi114.stderr: 90.0% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T04:43:17.780 INFO:teuthology.orchestra.run.smithi114.stderr: 81.2% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T04:43:17.780 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T04:43:17.780 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-volume.log: 84.7% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T04:43:17.780 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mon.b.log: 76.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T04:43:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.log 2024-03-26T04:43:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T04:43:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log: /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T04:43:17.782 INFO:teuthology.orchestra.run.smithi177.stderr: 81.3% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T04:43:17.782 INFO:teuthology.orchestra.run.smithi177.stderr: 90.1% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T04:43:17.782 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T04:43:17.782 INFO:teuthology.orchestra.run.smithi177.stderr: 84.8% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T04:43:17.782 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mon.c.log: 75.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T04:43:17.782 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-26T04:43:17.782 INFO:teuthology.orchestra.run.smithi114.stderr: 90.3% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T04:43:17.792 INFO:teuthology.orchestra.run.smithi114.stderr: 90.2% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T04:43:17.800 INFO:teuthology.orchestra.run.smithi038.stderr: 90.1% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T04:43:17.801 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T04:43:17.852 INFO:teuthology.orchestra.run.smithi114.stderr: 92.5% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T04:43:17.854 INFO:teuthology.orchestra.run.smithi177.stderr: 92.3% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T04:43:17.867 INFO:teuthology.orchestra.run.smithi038.stderr: 88.0% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T04:43:17.994 INFO:teuthology.orchestra.run.smithi038.stderr: 91.6% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T04:43:18.081 INFO:teuthology.orchestra.run.smithi177.stderr: 94.2% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T04:43:18.082 INFO:teuthology.orchestra.run.smithi038.stderr: 94.2% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T04:43:18.082 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-26T04:43:18.083 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m0.319s 2024-03-26T04:43:18.083 INFO:teuthology.orchestra.run.smithi177.stderr:user 0m0.407s 2024-03-26T04:43:18.083 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m0.036s 2024-03-26T04:43:18.084 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-26T04:43:18.084 INFO:teuthology.orchestra.run.smithi038.stderr:real 0m0.325s 2024-03-26T04:43:18.084 INFO:teuthology.orchestra.run.smithi038.stderr:user 0m0.646s 2024-03-26T04:43:18.084 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m0.043s 2024-03-26T04:43:18.121 INFO:teuthology.orchestra.run.smithi114.stderr: 94.2% -- replaced with /var/log/ceph/56caa506-eb2a-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T04:43:18.123 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-26T04:43:18.123 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m0.362s 2024-03-26T04:43:18.123 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m0.438s 2024-03-26T04:43:18.123 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m0.040s 2024-03-26T04:43:18.123 INFO:tasks.cephadm:Archiving logs... 2024-03-26T04:43:18.124 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028/remote/smithi038/log 2024-03-26T04:43:18.125 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T04:43:18.250 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028/remote/smithi114/log 2024-03-26T04:43:18.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T04:43:18.339 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028/remote/smithi177/log 2024-03-26T04:43:18.340 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T04:43:18.429 INFO:tasks.cephadm:Removing cluster... 2024-03-26T04:43:18.429 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 --force 2024-03-26T04:43:18.613 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:43:18.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 --force 2024-03-26T04:43:18.906 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:43:19.018 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 56caa506-eb2a-11ee-95d0-87774f69a715 --force 2024-03-26T04:43:19.200 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: 56caa506-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:43:19.323 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T04:43:19.323 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T04:43:19.341 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T04:43:19.356 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T04:43:19.371 INFO:tasks.cephadm:Teardown complete 2024-03-26T04:43:19.371 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-26T04:43:19.380 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi038:/dev/vg_nvme/lv_1... 2024-03-26T04:43:19.381 DEBUG:teuthology.orchestra.run.smithi038:> sudo nvme disconnect -n lv_1 2024-03-26T04:43:19.529 INFO:teuthology.orchestra.run.smithi038.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-26T04:43:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:19.531 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi038:/dev/vg_nvme/lv_2... 2024-03-26T04:43:19.531 DEBUG:teuthology.orchestra.run.smithi038:> sudo nvme disconnect -n lv_2 2024-03-26T04:43:19.672 INFO:teuthology.orchestra.run.smithi038.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-26T04:43:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:19.674 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi038:/dev/vg_nvme/lv_3... 2024-03-26T04:43:19.674 DEBUG:teuthology.orchestra.run.smithi038:> sudo nvme disconnect -n lv_3 2024-03-26T04:43:19.838 INFO:teuthology.orchestra.run.smithi038.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-26T04:43:19.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:19.840 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi038:/dev/vg_nvme/lv_4... 2024-03-26T04:43:19.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo nvme disconnect -n lv_4 2024-03-26T04:43:19.980 INFO:teuthology.orchestra.run.smithi038.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-26T04:43:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:19.981 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T04:43:19.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/scratch_devs 2024-03-26T04:43:20.007 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi114:/dev/vg_nvme/lv_1... 2024-03-26T04:43:20.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo nvme disconnect -n lv_1 2024-03-26T04:43:20.148 INFO:teuthology.orchestra.run.smithi114.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-26T04:43:20.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:20.149 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi114:/dev/vg_nvme/lv_2... 2024-03-26T04:43:20.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo nvme disconnect -n lv_2 2024-03-26T04:43:20.290 INFO:teuthology.orchestra.run.smithi114.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-26T04:43:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:20.292 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi114:/dev/vg_nvme/lv_3... 2024-03-26T04:43:20.292 DEBUG:teuthology.orchestra.run.smithi114:> sudo nvme disconnect -n lv_3 2024-03-26T04:43:20.453 INFO:teuthology.orchestra.run.smithi114.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-26T04:43:20.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:20.455 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi114:/dev/vg_nvme/lv_4... 2024-03-26T04:43:20.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo nvme disconnect -n lv_4 2024-03-26T04:43:20.584 INFO:teuthology.orchestra.run.smithi114.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-26T04:43:20.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:20.586 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-26T04:43:20.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/scratch_devs 2024-03-26T04:43:20.612 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi177:/dev/vg_nvme/lv_1... 2024-03-26T04:43:20.613 DEBUG:teuthology.orchestra.run.smithi177:> sudo nvme disconnect -n lv_1 2024-03-26T04:43:20.757 INFO:teuthology.orchestra.run.smithi177.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-26T04:43:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:20.758 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi177:/dev/vg_nvme/lv_2... 2024-03-26T04:43:20.758 DEBUG:teuthology.orchestra.run.smithi177:> sudo nvme disconnect -n lv_2 2024-03-26T04:43:20.911 INFO:teuthology.orchestra.run.smithi177.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-26T04:43:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:20.913 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi177:/dev/vg_nvme/lv_3... 2024-03-26T04:43:20.913 DEBUG:teuthology.orchestra.run.smithi177:> sudo nvme disconnect -n lv_3 2024-03-26T04:43:21.071 INFO:teuthology.orchestra.run.smithi177.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-26T04:43:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:21.073 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi177:/dev/vg_nvme/lv_4... 2024-03-26T04:43:21.073 DEBUG:teuthology.orchestra.run.smithi177:> sudo nvme disconnect -n lv_4 2024-03-26T04:43:21.219 INFO:teuthology.orchestra.run.smithi177.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-26T04:43:21.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:21.221 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-26T04:43:21.221 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/scratch_devs 2024-03-26T04:43:21.251 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T04:43:21.260 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T04:43:21.261 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:43:21.263 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:43:21.266 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:43:21.276 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:43:21.278 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:43:21.280 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:43:21.280 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-03-26T04:43:21.280 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 +538us[ +453us] +/- 83ms 2024-03-26T04:43:21.280 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -509us[ -594us] +/- 50ms 2024-03-26T04:43:21.280 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +277us[ +192us] +/- 84ms 2024-03-26T04:43:21.280 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:43:21.282 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:43:21.283 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-03-26T04:43:21.283 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 +539us[ +539us] +/- 83ms 2024-03-26T04:43:21.283 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv02.front.sepia.ceph.com 3 6 377 60 -508us[ -583us] +/- 50ms 2024-03-26T04:43:21.283 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +292us[ +292us] +/- 84ms 2024-03-26T04:43:21.283 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:43:21.307 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:43:21.311 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:43:21.312 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-03-26T04:43:21.312 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 +547us[ +464us] +/- 83ms 2024-03-26T04:43:21.312 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -511us[ -594us] +/- 50ms 2024-03-26T04:43:21.312 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +290us[ +207us] +/- 84ms 2024-03-26T04:43:21.312 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:43:21.313 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T04:43:21.321 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T04:43:21.322 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T04:43:21.333 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T04:43:21.411 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T04:43:21.481 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T04:43:21.556 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:43:21.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:21.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:43:21.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:21.618 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:43:21.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:21.650 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T04:43:21.661 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T04:43:21.672 INFO:teuthology.task.internal:Duration was 731.728249 seconds 2024-03-26T04:43:21.672 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T04:43:21.683 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T04:43:21.684 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T04:43:21.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T04:43:21.689 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T04:43:21.729 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:43:21.731 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:43:21.735 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:43:22.178 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T04:43:22.178 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T04:43:22.178 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T04:43:22.199 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-03-26T04:43:22.199 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T04:43:22.240 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-03-26T04:43:22.241 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T04:43:22.263 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T04:43:22.263 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T04:43:22.266 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T04:43:22.283 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T04:43:22.334 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T04:43:22.335 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T04:43:22.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T04:43:22.351 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T04:43:22.532 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T04:43:22.542 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T04:43:22.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T04:43:22.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T04:43:22.596 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T04:43:22.622 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T04:43:22.632 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:43:22.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:43:22.638 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:43:22.657 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-03-26T04:43:22.661 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-03-26T04:43:22.687 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-03-26T04:43:22.714 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:43:22.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:22.740 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:43:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:22.755 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:43:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:43:22.769 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T04:43:22.779 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T04:43:22.780 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028/remote/smithi038 2024-03-26T04:43:22.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T04:43:22.819 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028/remote/smithi114 2024-03-26T04:43:22.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T04:43:22.856 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028/remote/smithi177 2024-03-26T04:43:22.856 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T04:43:22.893 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T04:43:22.893 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T04:43:22.896 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T04:43:22.898 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T04:43:22.941 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T04:43:22.953 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T04:43:22.953 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T04:43:22.964 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T04:43:22.964 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T04:43:22.966 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T04:43:22.968 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T04:43:22.978 INFO:teuthology.orchestra.run.smithi038.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 04:43 /home/ubuntu/cephtest 2024-03-26T04:43:22.980 INFO:teuthology.orchestra.run.smithi114.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 04:43 /home/ubuntu/cephtest 2024-03-26T04:43:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 04:43 /home/ubuntu/cephtest 2024-03-26T04:43:22.996 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T04:43:23.007 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T04:43:23.037 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T04:43:23.072 DEBUG:teuthology.nuke:shortname: smithi038 2024-03-26T04:43:23.072 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:43:23.096 DEBUG:teuthology.nuke:shortname: smithi114 2024-03-26T04:43:23.096 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:43:23.101 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:22:39.310990', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDBkrJvXZfRGk5fUXdlzv0HcvGm4QDe2mWwho5eKhM5SCQUzLQRsV9kS7aEF3FbKkYTpqhBhMBo9YfcN+AST1ug='} 2024-03-26T04:43:23.119 DEBUG:teuthology.nuke:shortname: smithi177 2024-03-26T04:43:23.119 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:43:23.123 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:22:39.312576', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJe7DArtM43IOtvCVGqhbM4xapv46UVf0TwxMzz1WtnDgCd0pCxvOFxBNsLc46+q8QwqDPdzseALl9AlDiP1IU='} 2024-03-26T04:43:23.147 INFO:teuthology.orchestra.console.smithi038:Power off 2024-03-26T04:43:23.147 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T04:43:23.157 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622028', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:22:39.314039', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFWqv+3na21AEw5KjMzBAXaEMz65jfbvuqS2maLaV/dTRixKIMWs8HGEcX0o8QeKdxMJ8dSdKvC2FTyWLk6naBg='} 2024-03-26T04:43:23.172 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off 2024-03-26T04:43:23.172 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:43:23.191 INFO:teuthology.orchestra.console.smithi114:Power off 2024-03-26T04:43:23.191 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T04:43:23.201 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-03-26T04:43:23.213 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2024-03-26T04:43:23.214 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:43:23.228 INFO:teuthology.orchestra.console.smithi177:Power off 2024-03-26T04:43:23.228 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T04:43:23.237 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-26T04:43:23.250 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-03-26T04:43:23.251 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:43:23.276 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-26T04:43:27.203 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:43:27.238 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:43:27.277 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:43:27.313 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-03-26T04:43:27.348 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-26T04:43:27.385 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-26T04:43:31.314 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:43:31.349 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:43:31.387 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:43:31.425 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off 2024-03-26T04:43:31.458 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-26T04:43:31.496 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-03-26T04:43:31.526 INFO:teuthology.orchestra.console.smithi038:Power off completed 2024-03-26T04:43:31.597 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-03-26T04:43:35.461 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:43:35.570 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2024-03-26T04:43:35.671 INFO:teuthology.orchestra.console.smithi114:Power off completed 2024-03-26T04:43:35.775 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} duration: 731.7282485961914 failure_reason: '"2024-03-26T04:38:59.972242+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-26T04:43:35.775 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T04:43:35.855 INFO:teuthology.run:FAIL